Skip to content

feat(spec-decode): multi-pass TTT training for EAGLE-3 drafts (Megatron) - #3699

Open
MarkXCloud wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
MarkXCloud:spec
Open

MarkXCloud wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
MarkXCloud:spec

Conversation

@MarkXCloud

@MarkXCloud MarkXCloud commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Extends the single-pass EAGLE-3 draft co-training path on the Megatron policy backend to multi-pass TTT training (policy.draft.ttt_steps > 1), co-trained with the policy during GRPO and refit into vLLM alongside the policy weights. The Automodel backend's TTT path landed in #3960; both backends now read the same ttt_steps / ttt_step_loss_decay config.

Issues

Part of #3698 (1/5). Stacked series: this PR → #3712 (DFlash) → #3713 (DSpark) → #4059 (sequence packing + CP) → #4060 (PP).

Usage

policy:
  draft:
    enabled: true
    model_name: <eagle3 draft ckpt or null>
    loss_weight: 1.0        # as in the runs above
    ttt_steps: 3              # default; one pass per served speculative token; 1 = single-pass
    ttt_step_loss_decay: 1.0  # pass d weighted by decay**(d-1); 1.0 = uniform
  megatron_cfg:
    sequence_parallel: false  # required for ttt_steps > 1
  sequence_packing:
    enabled: false            # required for ttt_steps > 1

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

Rebased onto main after the typed speculator contract (#3701), streaming soft-CE (#3703), split-step normalization (#3706), draft optimizer overrides (#3707) and Automodel draft co-training (#3960) landed; the pieces those cover were dropped from this PR. Main pieces:

  • Docs: a Multi-pass TTT section in docs/guides/eagle3-speculative-decoding.md.
  • Multi-pass TTT (EagleModel.forward_ttt): sequential draft passes with a two-part trunk+branch flash attention (exact joint-LSE merge), inference-aligned per-pass RoPE offsets, non-detached hidden-state chain between passes.
  • Multi-pass loss: DraftTTTCrossEntropyLossFn — the multi-pass sibling of DraftCrossEntropyLossFn, which stays the ttt_steps == 1 loss (including the packed layout) — slices each pass (pass d predicts x_{i+d+1}) and feeds it to the streaming soft-CE as one DraftLossStats bin, weighted by ttt_step_loss_decay ** (d - 1) and normalized by per-pass global valid counts (DP-only all-reduce).
  • Config: the Megatron path now reads Eagle3DraftConfig.ttt_steps / ttt_step_loss_decay, which feat(automodel): speculative decoding draft co-training #3960 added for the Automodel path (default ttt_steps: 3; this PR validates ttt_steps >= 1 and a finite, non-negative decay). On Megatron, ttt_steps > 1 rejects sequence packing (until feat(spec-decode): context parallelism and sequence packing for draft co-training (Megatron) #4059) and sequence parallelism, and begin_train_step rejects it on the split train-step path (its DraftStepState takes a single loss bin). Megatron configs that leave ttt_steps unset now train 3 passes, so the single-pass eagle3 recipe pins ttt_steps: 1. ttt_steps == 1 routes to the existing single-pass forward and loss; it is not a one-pass run of the TTT loss.
  • Aux-layer capture: the hidden-state capture taps the aux layers the draft was built for, resolved rank-independently so PP stages agree on the P2P schedule.
  • vLLM refit safety: un-share the drafter lm_head before loading a trained head (vLLM's share-on-identical-weights alias would let the draft refit silently overwrite the serving target's head).
  • Nightly: grpo-qwen3-1.7b-1n8g-megatron-eagle3-ttt3, the eagle3 nightly with ttt_steps: 3. Beyond the single-pass checks it requires the pass-3 loss to fall below 0.8x its step-1 value (observed 0.59-0.60x) and the last-5 acceptance length to stay above 2.05 (step 1: 1.90; observed 2.26 and 2.29 in two runs). Nightly GPU-hour cap 5203 → 5227.

Smoke-tested on a Qwen3-8B GRPO math recipe. Both eagle3 nightlies (single-pass and ttt3) pass end to end on a vLLM 0.29 container (50 steps, 1n8g), re-run on the current head.

A suspected TP grad all-reduce gap on the (unexercised) split-API train-step path was found during this work; it is tracked separately with its regression test in #3716 rather than bundled here.

Evaluation verification

  • Model: Qwen3-8B
  • Benchmark: AIME24
  • Purple line: Eagle3 pretrained by AngelSlim/Qwen3-8B_eagle3
  • Pink line: Eagle3 w/ ttt=3, trained from random initialization
  • Blue line: no speculative decoding
image

Perf:
image

ttt_steps=3 vs ttt_steps=1

Same init and recipe in each pair; only ttt_steps differs.

Qwen3-8B GRPO math, randomly initialized EAGLE-3 draft (TP2, loss_weight: 1.0, draft lr 1e-4, 3 speculative tokens; mean over steps 169-178):

ttt1 ttt3
vllm/spec_acceptance_length 1.65 1.93
vllm/spec_acceptance_rate-pos-{1,2,3} 0.60 / 0.05 / 0.00 0.54 / 0.27 / 0.13
policy training s/step 24.0 25.9
generation tok/s 19.0k 21.6k

Single-pass training never sees the draft's own outputs, so positions 2-3 barely train; ttt3 trains every served position. Measured on this PR's initial revision: the loss has since moved onto main's streaming soft-CE (#3703), while the TTT forward and attention are unchanged; the ttt3 run's first 25 steps predate the inference-aligned RoPE offset. Peak memory was not logged in these runs.

Qwen3-1.7B, the two eagle3 nightlies at this head (AngelSlim/Qwen3-1.7B_eagle3 init, loss_weight: 1.0, no draft optimizer, 1n8g; mean over the last 5 of 50 steps):

ttt1 ttt3
vllm/spec_acceptance_length 2.24 2.29
vllm/spec_acceptance_rate-pos-{1,2,3} 0.65 / 0.38 / 0.21 0.65 / 0.40 / 0.24
policy training s/step 0.45 0.49
total s/step 8.44 8.74
peak train memory (max_memory_allocated, GiB) 11.19 11.40
generation tok/s 8.36k 8.79k

From a pretrained drafter the gap is small at 50 steps; this pair is the cost check. Peak memory is the max over ranks of torch.cuda.max_memory_allocated() across train(), from a temporary probe that is not part of this PR.

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Aug 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@MarkXCloud MarkXCloud changed the title feat(spec-decode): co-train EAGLE-3 TTT and DFlash/DSpark draft models during GRPO feat(spec-decode): DFlash/DSpark and EAGLE-3 TTT draft co-training Aug 18, 2026
@MarkXCloud MarkXCloud changed the title feat(spec-decode): DFlash/DSpark and EAGLE-3 TTT draft co-training feat(spec-decode): DFlash/DSpark and EAGLE-3 TTT co-training (Megatron) Aug 18, 2026
@MarkXCloud MarkXCloud changed the title feat(spec-decode): DFlash/DSpark and EAGLE-3 TTT co-training (Megatron) feat(spec-decode): multi-pass TTT training for EAGLE-3 drafts (Megatron) Aug 19, 2026
@MarkXCloud
MarkXCloud force-pushed the spec branch 2 times, most recently from ef0fe42 to e93d5d8 Compare September 2, 2026 08:09

@zpqiu zpqiu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the multi-pass TTT draft-training path, focused on correctness and CI state. This was a deep pass — six reviewers across the RL code, Megatron/flash-attn call chains, the vLLM refit path, and the test suite, with findings adversarially challenged before landing here. Roughly 35 raw findings were reduced to the 12 comments below; a dozen were dropped as unreachable, pre-existing, or style, and nine were withdrawn by their own authors under challenge.

The hardest part of this PR is correct. The trunk+branch joint-LSE attention was verified three independent ways: analytically against pinned flash-attn 2.8.1 source (the softmax_lse natural-log base and the FA-backward-with-joint-(out, lse) trick both check out), empirically on an H100 against a dense fp32 full-attention reference (fwd 3.09e-03, worst gradient 5.44e-03 in bf16), and by your own tests, which compare against modelopt's set_multi_step_attention_mask as an external oracle. ChunkedDistributedCrossEntropy was separately confirmed correct on two GPUs against a full-vocab reference. The per-pass RoPE offsets, GQA head grouping, and DP-only count all-reduce all check out too. That's a stronger evidence base than most PRs this size carry, and the AIME24 three-arm plot is a genuinely useful convergence check.

The main ask is CI. Seven tests currently fail and pyrefly reports a new error. Since pyproject.toml:723 puts -x in addopts, a local run stops at the first one — --maxfail=100 shows them all, which may be why this wasn't visible. Three of the failures are simply the Megatron-LM submodule moving under you (main bumped it in a0edb27 and upstream fixed the underlying issue), so the fix there is deleting code rather than writing it.

Two items are worth a look even though they aren't reachable on the shipped recipes — the vLLM drafter lookup and the PP aux-layer divergence — because both fail silently or fatally rather than loudly, and both have small fixes.

Documentation comments are deliberately omitted, since that box is unchecked. One thing that would round out the evidence when you're ready for review: spec-decode acceptance rate (mean accepted tokens/step) for ttt_steps=3 vs the pretrained drafter, since that's the headline metric for the rollout speedup the perf plot is showing.

Generated by Claude Code

Comment thread nemo_rl/models/megatron/draft/utils.py Resolved
return
import copy

draft_model.lm_head = copy.deepcopy(target_lm_head)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/generation/vllm/vllm_backend.py:639

pyrefly fails on this line, and while you're here there's a subtlety worth guarding.

The typecheck (a pre-commit hook, pass_filenames: false) reports a new error introduced by this line — main has 0, this branch has 1:

ERROR Argument `Any | None` is not assignable to parameter `value` with type `Module | Tensor`
  in function `torch.nn.modules.module.Module.__setattr__` [bad-argument-type]
  --> nemo_rl/models/generation/vllm/vllm_backend.py:639

An isinstance(target_lm_head, torch.nn.Module) narrow above the storage check fixes it.

The separate subtlety: torch.nn.Parameter.__deepcopy__ rebuilds a bare Parameter, dropping attributes vLLM attaches via set_weight_attrs (vocab_parallel_embedding.py#L58). Reproduced on torch 2.11.0+cu130:

ORIGINAL attrs: ['weight_loader', 'input_dim', 'output_dim']
DEEPCOPY attrs: []

load_weights resolves the loader off the param (models/utils.py#L277) and would fall back to default_weight_loader, which can't shard and asserts under TP>1. The deep-copied head also carries the target's org_vocab_size, which _trim_vocab_padding then reads for the draft.

Worth noting this is currently unreachable, so it's a latent issue rather than something you're hitting: NeMo-RL's own has_own_lm_head patch (nemo_rl/models/generation/vllm/patches.py:161) sets draft_vocab_size != vocab_size, and both shipped drafters are truncated-vocab (AngelSlim/Qwen3-{1.7B,8B}_eagle3: vocab_size=151936, draft_vocab_size=32000), so vLLM never shares the head and this method early-returns. It would bite a full-vocab drafter. Building a fresh module and copy_()-ing the data is more robust than deepcopy-plus-reattach.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the isinstance narrow (the file is pyrefly-clean now). For the latent attrs issue I kept deepcopy but re-bind the source params' __dict__ onto the copies, with a test pinning weight_loader survival — reattach avoids reproducing ParallelLMHead's ctor args.

Comment thread nemo_rl/models/megatron/draft/eagle.py Resolved
Comment thread nemo_rl/models/megatron/train.py Outdated
if draft_model is not None:
configured_aux_layers = getattr(
getattr(draft_model, "config", None),
"eagle_aux_hidden_state_layer_ids",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nemo_rl/models/megatron/train.py:295

Under PP > 1 this resolves to a different aux-layer list on each stage.

draft_model is attached only to the post-process chunk, so find_draft_owner_chunk returns None on every non-last stage (nemo_rl/models/megatron/draft/utils.py:975) and aux_layer_indices is None there. HiddenStateCapture.__init__ then falls back to get_eagle3_aux_hidden_state_layers(num_layers) (nemo_rl/models/megatron/draft/hidden_capture.py:66), while the last stage uses the configured ids. _gather_distributed iterates that list posting one send/recv per index (nemo_rl/models/megatron/draft/hidden_capture.py:251), so two different lists means mismatched counts — a wrong-tensor match that shapes won't catch, then a desync.

This is not reachable on the shipped recipes: neither AngelSlim/Qwen3-1.7B_eagle3 nor Qwen3-8B_eagle3 carries eagle_aux_hidden_state_layer_ids, so build_draft_model falls back to the same get_eagle3_aux_hidden_state_layers(model_provider.num_layers) and the lists coincide. The reachable trigger is the documented knob: policy.draft.aux_layer_indices: [...] with pipeline_model_parallel_size > 1 and packing off (which the guard at nemo_rl/models/policy/lm_policy.py:146 permits).

Flagging honestly: this is a static trace — I did not run it on two PP stages. Resolving the indices from a rank-independent source (or all-gathering them over the PP group and asserting agreement) would remove the divergence.

Minor, same block: the nested getattr(getattr(draft_model, "config", None), ...) can be a direct read since build_draft_model always sets that attribute.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed at the root — new resolve_draft_aux_layer_ids() resolves the list rank-independently and is threaded into both forward paths, and build_draft_model uses the same resolver so the draft and the capture can't disagree. The nested getattr is a direct read now.

Comment thread nemo_rl/models/policy/__init__.py Resolved
import pytest
import torch

pytest.importorskip("vllm", reason="vllm_backend requires a vLLM install")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tests/unit/models/generation/test_vllm_draft_lm_head_unshare.py:30

This file never executes in CI — verified Running 0 items in this shard.

L0_Unit_Tests_Vllm_1.sh runs two passes: a base run that globs test_vllm*.py with --no-sync (vLLM is an optional extra, so importorskip skips the module), and a --extra vllm --vllm-only catch-all that collects only marker-carrying items. With importorskip but no marker, the file falls through both. test_nccl_reshard_backend.py is the exemplar — it pairs the marker with an importorskip, exactly this situation.

Verified fix takes it from 0 collected to 3 passing:

Suggested change
pytest.importorskip("vllm", reason="vllm_backend requires a vLLM install")
pytestmark = pytest.mark.vllm
pytest.importorskip("vllm", reason="vllm_backend requires a vLLM install")

Worth pairing with one assertion, since the current stubs would pass even with the weight_loader issue above present (draft_model is a SimpleNamespace, so nn.Module.__setattr__ is never exercised, and no ParallelLMHead is ever built): after un-sharing, assert hasattr(draft_model.lm_head.weight, "weight_loader"). The _load_draft_weights raise path is also untested — a pytest.raises there would have caught the PP case above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took the fix (marker + importorskip), added the weight_loader assertion, and added the _load_draft_weights raise-path tests you suggested.

Comment thread nemo_rl/algorithms/loss/wrapper.py Resolved

@seonjinn seonjinn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Read this branch merged with current main (clean merge, 59 files). @zpqiu has already covered the correctness of the joint-LSE trunk+branch attention and the CI state, so this review does not revisit either — everything below is either a config-plumbing defect or a collision with the drafter stack that is landing in parallel.

One framing note before the findings: CI on this PR runs only DCO, the submodule comment, semantic-pull-request and triage. There is no unit-test lane, so the seven failures reported from a local run will not surface here no matter how many times it is re-run, and none of the defects below are catchable by the checks currently on the PR.

Actionable

1. Multi-pass normalization is silently wrong on the split train-step path. megatron_policy_worker.py:1580 threads draft_ttt_steps=draft_ttt_steps into the split step but not global_draft_pass_counts, while the monolithic path passes both (:978 and :983). With ttt_steps > 1 on the split path the loss therefore receives global_draft_pass_counts=None and falls back to denominator = global_valid_toks (loss_functions.py:218-219) — the policy's token count, not the intended sum_d alpha_d * count_d. The draft gradient is scaled by the wrong denominator, by roughly the ratio of the two counts, with no error and no metric that would look obviously wrong.

Either compute the counts on the split path as well, or reject the combination outright. I would reject for now: raise NotImplementedError when ttt_steps > 1 and the split path is selected. That also becomes mandatory once #3706 lands — see item 5 of the follow-up comment.

2. ttt_steps: 0 is silently coerced to 1. int(draft_cfg.get("ttt_steps", 1) or 1) (train.py:659, and the same expression at megatron_policy_worker.py:872 and :1547) turns 0 into 1 because 0 is falsy, while -1 passes straight through to whatever fails first. A user who writes ttt_steps: 0 meaning "off" gets a single-pass run and no message. This wants one validated field rather than three copies of a defensive coercion — see item 2 of the follow-up comment.

3. The weight-decay-only draft optimizer config is rejected. DraftOptimizerConfigOverrideProvider.build_config_overrides (draft/utils.py:1328-1334) raises ValueError: [draft] draft (lr, min_lr) must differ from the policy's. whenever the effective (max_lr, min_lr) equals the policy's. Setting only draft.weight_decay leaves both LRs unset, so effective_max_lr/effective_min_lr fall back to the policy's values and the raise fires — even though the requested override (start_wd/end_wd) is perfectly distinct and would work. Giving the drafter a different weight decay at the same LR is a reasonable thing to want and is currently unreachable.

The raise exists because the group's resume identity was assumed to be just (max_lr, min_lr). It is not, any more — see item 1 of the follow-up comment, which is also why I would delete this code path rather than patch the condition.

4. ttt_pass_weights length is validated per microbatch, not at startup. The check lives in DraftTTTCrossEntropyLossFn.__call__ (loss_functions.py:191-195), so a mismatched list is caught on the first forward of the first microbatch rather than at config parse. The length is fully known at config time. Same for the global_draft_pass_counts shape check right below it, which is derived from the same ttt_steps.

5. The forward-side pass count and the loss-side pass count come from different places. The forward selects forward_ttt from the draft_ttt_steps function parameter, which defaults to 1 (train.py:233, used at :369), while DraftLossWrapper is constructed with ttt_steps read from self.cfg["draft"] (train.py:659, :683). Both call sites in this PR pass the kwarg, so the two agree today. But the default makes the divergence silent-by-construction for the next caller: omit the kwarg with ttt_steps: 3 in the config and the forward produces single-pass output while the loss asks for student_logits_by_pass, i.e. a KeyError some distance from the cause. Deriving both from one config object removes the trap.

6. raise ValueError(f"Unknown draft_method ...") is unreachable. method is read as draft_cfg.get("method", "eagle3") (train.py:658) but is declared in no schema and set in no YAML in the repo, so the only reachable value is the default. This is a symptom of the selector duplication rather than a bug on its own — see item 4 of the follow-up comment.

Nit

self.seq_chunk_size defaults to 4096 in DraftTTTCrossEntropyLossFn.__init__ (loss_functions.py:155), but train.py:668-671 only forwards loss_seq_chunk_size when ttt_steps > 1. In the single-pass case the key is accepted by the config and then ignored. Harmless today because DraftCrossEntropyLossFn does not chunk, but it reads as a knob that works everywhere.

Verified clean

The guard placement is right: ttt_steps > 1 with sequence_packing is rejected early at lm_policy.py:176-188, where the user sees it at setup, and CP / sequence-parallel for ttt_steps > 1 are rejected in build_draft_model (draft/utils.py:1471-1481). The per-pass metric denominators are correctly the global per-pass counts rather than per-microbatch means (loss_functions.py:247-259) — the comment there explaining the ~512x inflation at gbs 512 is worth keeping.

ChunkedDistributedCrossEntropy's memory argument holds: saving the bf16 logits references and recomputing the chunk softmaxes in backward is the right trade when several TTT passes stack before a shared backward, and it mirrors ChunkedDistributedLogprob. My only objection to it is that #3703 already ships the same kernel — see the follow-up comment.

Method note

Nothing except lint was executed: torch is not importable in the review environment, so all of the above is read from source on the merged tree. ruff 0.9.9 check and format --check are both clean on the merged tree (640 files formatted), so the reported CI failures are test failures, not lint. The submodule pin was resolved through 3rdparty/Megatron-Bridge-workspace/Megatron-Bridge at b11414c71b15e54d333eb49346ed199f20fa9021.

@seonjinn seonjinn left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Duplicate submission — same content as the review directly above. Sorry for the noise.)

@seonjinn

seonjinn commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Integration suggestions for the DFlash/DSpark drafter stack

Thanks again for this work. Would it be okay if I leave a few integration-focused suggestions here?

I am happy for your team to continue owning the EAGLE-3 TTT implementation, and I do not intend to propose changes to the core algorithm — the TTT loss, the joint-LSE trunk+branch attention, and the multi-pass schedule are yours. What I would like to align on is the surrounding plumbing: the typed draft config, the loss-statistics interface, and the optimizer/refit structures, so that this integrates cleanly with the DFlash/DSpark online-training PRs landing alongside it (#3701 → #3703 → #3706 → #3707 → #3715 → #3723 → #3711 → #3726 → #3744 → #3750 → #3757).

Concretely, the four conventions I would suggest converging on:

  • a typed draft config with a shared selector, speculator_type: eagle3 rather than a separate method key
  • EAGLE-specific fields declared on Eagle3DraftConfig (ttt_steps, ttt_pass_weights, loss_seq_chunk_size), so they are validated once at parse time
  • a common draft loss-statistics interface, DraftLossStats with one bin per pass, which the TTT reduction maps onto exactly
  • consistent optimizer and refit structures, draft.optimizer.* and the shared _load_draft_weights contract

On timing, since none of the stack has landed yet. I am not asking you to depend on unmerged branches, and this PR should not be held waiting on them. The point of raising it now is that both sides can still agree cheaply on a shared shape, rather than discovering it as a conflict at merge time. The ordinary rule applies: whichever lands first stays, the second rebases. The items below are written for the case where the stack lands first, since that is the current plan of record — if it slips and this PR lands first, the same items apply in reverse and I will carry them on the stack side instead. Either way, most of the mechanical work below is work I am happy to do rather than hand to you; tell me which parts you would rather own.

In practice, only item 4 is worth doing before the stack merges. The rest is agreement-in-principle.

I have written these as concrete diffs rather than a neutral comparison, because a comparison leaves the decision unmade — not because the choice is settled. If you disagree on any of them, that is exactly the discussion worth having before either side merges.

Everything cited was read from this branch merged with current main, and from the stack branches.


1. Optimizer overrides: converge on #3707's structure

Both PRs add a function with the same name, wired into the same call:

  • here: build_draft_optimizer_override_provider at nemo_rl/models/megatron/draft/utils.py:1339, wired at nemo_rl/models/megatron/setup.py:2063
  • feat(draft): support optimizer schedule overrides #3707: same name at nemo_rl/models/megatron/draft/optimizer.py:82, wired at setup.py:2029

Neither is in main — git show origin/main:nemo_rl/models/megatron/draft/utils.py has no such symbol, and neither does main's setup.py. So this is a genuine duplicate, and the second one to merge gets a conflict in both files.

Suggestion: use #3707's provider instead of the one here. That means removing, from draft/utils.py:

  • _PARAM_GROUP_IDENTIFIER_EXTRA_KEYS (:1263)
  • extend_param_group_identifier_keys_for_resume (:1266)
  • DraftOptimizerConfigOverrideProvider (:1294-1336)
  • build_draft_optimizer_override_provider (:1339-1360)

and the wiring at setup.py:2063 plus the import at setup.py:245. Then delete tests/unit/models/megatron/test_draft_optimizer_overrides.py, which is the file already reported as carrying three failures.

Three reasons the stack's version is the one to keep, beyond merge order:

  • The monkeypatch is already dead. As noted separately above, param_group_identifier_keys is now built from ParamGroupOverride.__annotations__ upstream and already carries max_lr/min_lr/start_wd/end_wd. extend_param_group_identifier_keys_for_resume is a no-op against the pinned submodule. feat(draft): support optimizer schedule overrides #3707 never needed it.
  • The distinctness raise goes with it. The ValueError at draft/utils.py:1331 requiring (lr, min_lr) != (policy lr, policy min_lr) only exists because the identifier tuple used to be too narrow. It is what rejects the weight-decay-only config flagged in the separate comment above. feat(draft): support optimizer schedule overrides #3707 has no equivalent and does not need one, so deleting the patch deletes the bug.
  • The selector is more robust. This PR matches parameters by module path, ParamKey(name="*draft_model.*") (:1335). main already tags every draft parameter — DRAFT_GRAD_NORM_GROUP = "draft" at draft/utils.py:1239, stamped at :1531, registered as a separate grad-norm group at :1254 — and feat(draft): support optimizer schedule overrides #3707's _is_draft_parameter reads that tag. To be clear, the glob is not broken today: the leading * absorbs the module. prefixes that DDP and Float16Module add, and the draft is attached as exactly that attribute at setup.py:1671. The problem is that it is a second, independent answer to a question main already answers, and the two drift silently — rename the attachment attribute, or reuse the provider for the DFlash/DSpark bodies that feat(draft): add projected soft CE for DFlash #3715/feat(draft): add tensor-parallel DFlash body #3723/feat(draft): add DSpark auxiliary heads #3711 add, and the glob selects nothing while the grad-norm grouping keeps working.

One thing to carry over rather than drop. The two disagree on what min_lr defaults to, and this is a real behavioural difference, not a code-organization one:

  • here (draft/utils.py:1314-1320): with lr set and min_lr unset, min_lr follows the draft LR, so the drafter holds a constant LR while the policy decays. The comment says that is deliberate.
  • feat(draft): support optimizer schedule overrides #3707 (draft/optimizer.py:52-54): min_lr falls back to the policy's min_lr, so the drafter decays on the policy's floor.

If the constant-LR default is what your AIME24 runs used, that is worth saying on #3707 before it merges, because rebasing onto #3707 silently changes it. Both behaviours stay reachable either way by setting min_lr explicitly — it is only the default that differs.

2. EAGLE fields: declare them on Eagle3DraftConfig

This PR adds ttt_steps, ttt_pass_weights and loss_seq_chunk_size to the DraftConfig TypedDict (nemo_rl/models/policy/__init__.py:515-523). #3701 deletes that TypedDict outright and replaces it with PolicyConfig.draft: NotRequired[Eagle3DraftConfig] (policy/__init__.py:568), a pydantic model in nemo_rl/models/policy/draft_config.py.

Suggestion: declare the three keys as fields on Eagle3DraftConfig. Roughly:

    ttt_steps: Annotated[int, Field(ge=1)] = 1
    ttt_pass_weights: list[float] | None = None
    loss_seq_chunk_size: Annotated[int, Field(gt=0)] | None = None

    @model_validator(mode="after")
    def validate_ttt(self) -> Self:
        if self.ttt_pass_weights is not None and len(self.ttt_pass_weights) != self.ttt_steps:
            raise ValueError(
                f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, "
                f"got {len(self.ttt_pass_weights)}"
            )
        return self

This is worth doing for its own sake, not just for the merge. It fixes three things reported separately above in one place:

  • Field(ge=1) replaces int(draft_cfg.get("ttt_steps", 1) or 1), which silently turns ttt_steps: 0 into 1 while -1 raises. Both then route through one validator.
  • the ttt_pass_weights length check currently runs per-microbatch inside DraftTTTCrossEntropyLossFn.__call__, though the length is known at config time; as a validator it fails at startup.
  • pydantic defaults remove the five hidden ttt_steps defaults at train.py:659, draft/utils.py:1471, lm_policy.py:178, megatron_policy_worker.py:871 and :1548, which is what lets a caller omit the forward-side draft_ttt_steps and produce the KeyError: 'student_logits_by_pass' reported above.

If the flat lr/min_lr/weight_decay keys at policy/__init__.py:527-529 survive item 1, note that Eagle3DraftConfig is declared extra="allow" (draft_config.py:36) with no unknown-key check, so those three would be accepted and silently ignored rather than rejected — a user asking for a 50x draft LR would get 1x with no error and no failing test. Deleting them per item 1 is what closes that.

3. Draft-config reads: subscript to attribute access

Once draft is an Eagle3DraftConfig, cfg["draft"]["x"] raises TypeError: 'Eagle3DraftConfig' object is not subscriptable and .get(...) raises AttributeError. This PR adds new subscript sites on top of the ones main already has:

  • nemo_rl/models/megatron/train.py:657-668 (draft_cfg = self.cfg["draft"], then .get("method"), .get("ttt_steps"), .get("ttt_pass_weights"), .get("loss_seq_chunk_size"), draft_cfg["loss_weight"])
  • nemo_rl/models/policy/workers/megatron_policy_worker.py:871-873 and :1547-1548
  • nemo_rl/models/policy/lm_policy.py:178
  • nemo_rl/models/megatron/draft/utils.py:1471

#3701 provides coerce_draft_config (draft_config.py:48) for exactly the boundary case where a caller hands in a plain mapping, so these become draft_cfg.ttt_steps after a single coercion at the entry point rather than defensive .get() at each site.

4. Selector: one key, speculator_type

This PR adds a second drafter selector: draft_method = draft_cfg.get("method", "eagle3") (train.py:658), threaded into DraftLossWrapper as draft_method= (:682). #3701 already declares the selector as speculator_type: Literal["eagle3"] = "eagle3" (draft_config.py:39) and dispatches through _SPECULATOR_FACTORIES (nemo_rl/models/megatron/draft/training.py:67-81), and #3715/#3723/#3711/#3726 extend that same Literal for DFlash and DSpark.

method is declared in no schema and set in no YAML, which is also why the raise ValueError(f"Unknown draft_method ...") below it is unreachable today.

That unreachability stops being harmless once the stack lands. #3757 adds examples/configs/recipes/llm/grpo-qwen3-8b-1n8g-megatron-dflash.yaml, where policy.draft.speculator_type: dflash. method is absent from that file, so draft_cfg.get("method", "eagle3") returns "eagle3" and this PR builds an EAGLE-3 loss for a DFlash draft — two selectors disagreeing, with the wrong one winning silently. Worth noting separately that speculator_type is also the name of a vLLM field (vllm_config.speculative_config.method) used in #3744/#3750/#3757; those are different namespaces, and #3757's guide states the vLLM method must match policy.draft.speculator_type. One NeMo-RL-side selector is the only way that stays legible.

Suggested change: delete the method read and the draft_method parameter. Multi-pass TTT is an EAGLE-3 feature, and speculator_type already says the speculator is eagle3, so ttt_steps > 1 is sufficient to select the multi-pass loss — which is what DraftLossWrapper.multi_pass (wrapper.py:290) already keys on. That leaves one selector for "which drafter" and one knob for "how many passes", and the multi-pass loss registers through the same factory the DFlash/DSpark work will use.

5. #3706 makes the split-path guard mandatory, and points at the eventual fix

Separately noted above that the split train-step API threads draft_ttt_steps but not global_draft_pass_counts (megatron_policy_worker.py:1546), and that a NotImplementedError is the right response for now. #3706 turns that from a good idea into a requirement: it routes split-microbatch draft statistics through a new DraftStepState that hard-rejects anything other than a single bin with unit weight (nemo_rl/models/megatron/draft/step_state.py:73-79). Multi-pass statistics are [ttt_steps] bins with per-pass weights alpha_d, so after #3706 a ttt_steps > 1 run on the split path fails with ValueError: current EAGLE draft step state requires exactly one bin instead of mis-scaling. Adding the guard here keeps that a clear message at setup rather than a confusing one mid-step.

Worth flagging the constructive half, since it affects how #3706 should be reviewed: the multi-pass shape is exactly what DraftLossStats's numerators/counts/weights triple was built for, with weights = alpha. The one-bin restriction in accumulate is a deliberate narrowing, not a structural limit. If multi-pass split-step support is wanted later, generalizing that guard to [ttt_steps] bins is the natural path and the denominator sum_d alpha_d * count_d falls out of _normalization_scale as written. Saying so on #3706 now would stop the narrowing from hardening into an assumption.

6. Loss statistics: converge on DraftLossStats

This is the third duplicate, and the one where the two implementations are closest.

  • here: ChunkedDistributedCrossEntropy at nemo_rl/distributed/model_utils.py:270 — "Soft-target cross entropy across TP-sharded vocab, chunked along sequence", saving only the bf16 logits references and recomputing the chunk softmaxes in backward. Called from DraftTTTCrossEntropyLossFn._per_token_soft_ce (nemo_rl/algorithms/loss/loss_functions.py:165-172) with the loss_seq_chunk_size knob.
  • feat(draft): stream vocab-parallel soft CE #3703: streaming_vocab_parallel_soft_ce at nemo_rl/algorithms/loss/draft.py:324, dispatching to _StreamingVocabParallelSoftCE (:213) with the same recompute-in-backward structure, plus a _CachedVocabParallelSoftCE fast path (:149) when the whole input fits in one tile, and the token_chunk_size knob.

Same math, same memory strategy, two copies. #3703's is the superset: it takes the mask, returns per-bin reductions, and tiles over flattened tokens rather than the sequence axis, so it does not degrade when the batch dimension carries the tokens.

Suggested change: drop ChunkedDistributedCrossEntropy and have the per-pass reduction return a DraftLossStats with one bin per TTT pass. The two reductions already line up term for term. In the loop at loss_functions.py:223-262, pass_sum is the mask-weighted CE sum for pass d, pass_counts[pass_index] is its global token count, weight is alpha_d, and the final scalar is

total / denominator  =  sum_d alpha_d * pass_sum_d  /  sum_d alpha_d * pass_count_d

which is exactly DraftLossStats.normalized(normalization_counts=global_counts) — that method computes (numerators * weights).sum() / ((counts * weights).sum() + 1e-8) (draft.py:42-58). So the per-pass call becomes

stats_d = streaming_vocab_parallel_soft_ce(
    student_logits=student_slice,
    teacher_logits=teacher_slice,
    mask=pass_mask,
    token_chunk_size=self.seq_chunk_size,
    tp_group=self.vocab_parallel_group,
    weights=student_slice.new_tensor([weight]),
)

and the per-pass stats stack into a single [ttt_steps]-bin DraftLossStats. metrics[f"draft_loss_pass_{d}"] reads off numerators[d] / counts[d], which is what the current code recomputes by hand.

Three things this buys beyond removing the duplicate:

  • It is the concrete form of item 5. The stacked stats are the [ttt_steps]-bin object DraftStepState would need, so multi-pass split-step support later becomes generalizing feat(draft): preserve split-step normalization parity #3706's one-bin guard rather than a new data path.
  • nemo_rl/distributed/model_utils.py stops growing a draft-specific loss kernel. feat(draft): stream vocab-parallel soft CE #3703 puts it in nemo_rl/algorithms/loss/draft.py, next to the thing that consumes it.
  • The torch.clamp(denominator, min=1.0) at loss_functions.py:262 becomes normalized's + 1e-8. Worth a deliberate decision rather than an accident of which file won: clamping at 1.0 silently rescales any step whose global draft token count is below 1, which is only the empty-batch case, but the two conventions disagree there and only one survives the merge.

One thing to check rather than assume: #3703's kernel expects mask to match the non-vocab dims of the logits and folds it into the numerator itself, while the current code multiplies the mask in afterwards at :244. pass_mask is already draft_pass_token_mask(token_mask, ttt_pass) * sample_mask.unsqueeze(-1), i.e. the right shape and semantics, so this should be a straight substitution — but the pass-1 shifted-slot behaviour is worth one parity test against the current implementation before deleting anything.

7. The rest of the stack, and where in the order this PR belongs

I also read the seven later PRs (#3715, #3723, #3711, #3726, #3744, #3750, #3757) against this branch. Two have zero file overlap with it (#3723, #3711) and #3726 reaches it only through a hunk #3715 already owns, so the surface is smaller than the PR count suggests. The three that matter are #3744/#3750 in vllm_backend.py, and #3757, which touches nine of the same files.

Merge-order recommendation: land this PR after #3757, i.e. twelfth. It conflicts materially with only #3701 and #3757, and #3757 is already last — so putting this one after it means rebasing ~700 production lines onto a settled tree, rather than forcing #3757's 121-file diff to absorb this one across nine files.

The thing most at risk of being silently lost: _unshare_draft_lm_head. Defined at vllm_backend.py:602-644, called at :666, gated on the weight stream containing an lm_head. #3744 rewrites _load_draft_weights wholesale and #3750/#3757 carry that same rewrite; none of the three has any counterpart, and all three delete the lines the call sits between. A routine "take theirs" conflict resolution therefore drops it and leaves the function as dead code with no caller. Per the docstring here, vLLM's SpecDecodeBaseProposer._maybe_share_lm_head aliases the drafter's lm_head to the target's, so refitting a trained EAGLE head writes through the alias into the serving target. This is the only place in the twelve PRs that addresses that aliasing, and it is reachable on exactly the configuration this PR targets — the DFlash/DSpark export blocklists strip lm_head, so the gate never fires for them, but EAGLE-3 co-training is precisely where it does. Please re-apply the call inside #3744's rewritten body when rebasing. (I have not read vLLM's own source for the aliasing; that half is from the docstring here.)

Three smaller carry-overs in the same area:

One new contract hole, from #3757. It adds defer_normalization to DraftLossWrapper and on that path calls self.draft_loss_fn.loss_stats(...), adding loss_stats to DraftCrossEntropyLossFn. DraftTTTCrossEntropyLossFn has only __call__, so multi-pass would AttributeError there. It looks unreachable on any shipped config today — the deferral comes from the split-step API, which #3757 demands only for context_parallel_size > 1, and eagle3 + CP is rejected — but item 6 is what closes it properly: once the per-pass reduction returns a DraftLossStats, loss_stats is that same object and the method is nearly free to add.

8. Mechanical


To restate the intent: none of this touches the TTT algorithm, and none of it is a blocker from my side. Items 1, 2, 3, 6 and 8 are mechanical, and I am glad to prepare them as a follow-up on top of whichever branch merges second — say the word and I will open it rather than adding to your queue. Items 4, 5 and 7 are the ones where I would value your judgement, particularly the min_lr default in item 1 (if your AIME24 runs relied on the constant draft LR, I would rather change #3707 than change this) and the _unshare_draft_lm_head carry-over in item 7, which I think is the single most valuable thing in this PR and the easiest to lose in a rebase.

@seonjinn seonjinn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two of the items from my earlier comments as applicable suggestions, so they are easier to take or reject on the spot. Both are standalone — neither depends on the drafter stack landing.

Comment on lines +1545 to +1547
draft_ttt_steps = (
int(self.cfg["draft"].get("ttt_steps", 1) or 1) if draft_enabled else 1
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finding 1: this path threads draft_ttt_steps into megatron_forward_backward (:1578) but never computes global_draft_pass_counts, unlike the monolithic path (:881, passed at :976). The draft loss then falls back to denominator = global_valid_toks — the policy token count rather than sum_d alpha_d * count_d — so the draft gradient is mis-scaled with no error and no metric that looks wrong.

Failing fast is the smaller change; computing the counts here would also be correct if you prefer that.

Suggested change
draft_ttt_steps = (
int(self.cfg["draft"].get("ttt_steps", 1) or 1) if draft_enabled else 1
)
draft_ttt_steps = (
int(self.cfg["draft"].get("ttt_steps", 1) or 1) if draft_enabled else 1
)
if draft_ttt_steps > 1:
raise NotImplementedError(
"policy.draft.ttt_steps > 1 is not supported on the split train-step "
"path: global_draft_pass_counts is not computed here, so the draft "
"loss would normalize by the policy token count instead of "
"sum_d alpha_d * count_d."
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted: the split train-step path now raises NotImplementedError for ttt_steps > 1, which #3706's single-bin DraftStepState requires anyway.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in abfd277

Comment thread nemo_rl/models/megatron/train.py Outdated
Comment on lines +657 to +683
draft_cfg = self.cfg["draft"]
draft_method = draft_cfg.get("method", "eagle3")
ttt_steps = int(draft_cfg.get("ttt_steps", 1) or 1)
# Ctor kwargs for the method's draft LossFn (uniform shape;
# DraftLossWrapper splats them into the selected class). Only
# the multi-pass class takes pass weights.
draft_loss_kwargs: dict[str, Any] = (
{"pass_weights": draft_cfg.get("ttt_pass_weights")}
if ttt_steps > 1
else {}
)
if ttt_steps > 1 and draft_cfg.get("loss_seq_chunk_size") is not None:
draft_loss_kwargs["seq_chunk_size"] = int(
draft_cfg["loss_seq_chunk_size"]
)
loss_fn_wrapped = DraftLossWrapper(
loss_fn=loss_fn_wrapped,
prepare_fn=prepare_loss_input_wrapped,
data_dict=data_dict,
loss_weight=float(self.cfg["draft"]["loss_weight"]),
loss_weight=float(draft_cfg["loss_weight"]),
draft_loss_kwargs=draft_loss_kwargs,
global_draft_pass_counts=global_draft_pass_counts,
vocab_parallel_rank=get_tensor_model_parallel_rank(),
vocab_parallel_group=get_tensor_model_parallel_group(),
context_parallel_group=get_context_parallel_group(),
draft_method=draft_method,
ttt_steps=ttt_steps,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things folded into one suggestion, both from the earlier comments:

  • ttt_steps: 0 is silently coerced to 1. int(... or 1) turns 0 into 1 because 0 is falsy, while -1 passes through to whatever fails first. Explicit validation instead.
  • The method selector. draft_cfg.get("method", "eagle3") is declared in no schema and set in no YAML, so "eagle3" is the only reachable value and the Unknown draft_method raise in DraftLossWrapper is dead. draft_method already defaults to "eagle3" in the wrapper signature (wrapper.py:268), so dropping the read here is behaviour-preserving today — and it avoids a second selector disagreeing with speculator_type once the DFlash/DSpark configs land.

Happy to drop the second half if you would rather keep method for now; the first half stands on its own.

Suggested change
draft_cfg = self.cfg["draft"]
draft_method = draft_cfg.get("method", "eagle3")
ttt_steps = int(draft_cfg.get("ttt_steps", 1) or 1)
# Ctor kwargs for the method's draft LossFn (uniform shape;
# DraftLossWrapper splats them into the selected class). Only
# the multi-pass class takes pass weights.
draft_loss_kwargs: dict[str, Any] = (
{"pass_weights": draft_cfg.get("ttt_pass_weights")}
if ttt_steps > 1
else {}
)
if ttt_steps > 1 and draft_cfg.get("loss_seq_chunk_size") is not None:
draft_loss_kwargs["seq_chunk_size"] = int(
draft_cfg["loss_seq_chunk_size"]
)
loss_fn_wrapped = DraftLossWrapper(
loss_fn=loss_fn_wrapped,
prepare_fn=prepare_loss_input_wrapped,
data_dict=data_dict,
loss_weight=float(self.cfg["draft"]["loss_weight"]),
loss_weight=float(draft_cfg["loss_weight"]),
draft_loss_kwargs=draft_loss_kwargs,
global_draft_pass_counts=global_draft_pass_counts,
vocab_parallel_rank=get_tensor_model_parallel_rank(),
vocab_parallel_group=get_tensor_model_parallel_group(),
context_parallel_group=get_context_parallel_group(),
draft_method=draft_method,
ttt_steps=ttt_steps,
draft_cfg = self.cfg["draft"]
raw_ttt_steps = draft_cfg.get("ttt_steps", 1)
ttt_steps = 1 if raw_ttt_steps is None else int(raw_ttt_steps)
if ttt_steps < 1:
raise ValueError(
f"policy.draft.ttt_steps must be >= 1, got {ttt_steps}"
)
# Ctor kwargs for the draft LossFn (uniform shape; DraftLossWrapper
# splats them into the selected class). Only the multi-pass class
# takes pass weights.
draft_loss_kwargs: dict[str, Any] = (
{"pass_weights": draft_cfg.get("ttt_pass_weights")}
if ttt_steps > 1
else {}
)
if ttt_steps > 1 and draft_cfg.get("loss_seq_chunk_size") is not None:
draft_loss_kwargs["seq_chunk_size"] = int(
draft_cfg["loss_seq_chunk_size"]
)
loss_fn_wrapped = DraftLossWrapper(
loss_fn=loss_fn_wrapped,
prepare_fn=prepare_loss_input_wrapped,
data_dict=data_dict,
loss_weight=float(draft_cfg["loss_weight"]),
draft_loss_kwargs=draft_loss_kwargs,
global_draft_pass_counts=global_draft_pass_counts,
vocab_parallel_rank=get_tensor_model_parallel_rank(),
vocab_parallel_group=get_tensor_model_parallel_group(),
context_parallel_group=get_context_parallel_group(),
ttt_steps=ttt_steps,
)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both done: ttt_steps is Annotated[int, Field(ge=1)] on Eagle3DraftConfig, and the method read and draft_method parameter are gone, so speculator_type is the only selector and ttt_steps > 1 picks the multi-pass loss.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in abfd277

@MarkXCloud

MarkXCloud commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor Author

@zpqiu Thanks for the review — all seven test failures and the pyrefly error are fixed in the updated commit; replied on each comment in-thread.

Test evidence (container run against the pinned submodules, --maxfail=100 to get past -x): default pass 32 passed / 1 skipped (the vllm file's importorskip in the mcore venv); --mcore-only 60 passed (incl. the guard test and the modelopt mask oracle); --vllm-only 6 passed; pyrefly clean on vllm_backend.py.

Acceptance-rate data (ttt_steps=3 vs the pretrained drafter) is being collected and will be posted here.

Could a maintainer post /ok to test 5ab691cfa05bfd3d54120f117b1315412591521b and attach a CI:L1 label? I don't have label permissions on this repo.

@zpqiu zpqiu added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Sep 7, 2026
@zpqiu

zpqiu commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

/ok to test 5ab691c

@MarkXCloud
MarkXCloud force-pushed the spec branch 2 times, most recently from 5e503aa to abfd277 Compare October 13, 2026 07:22
@zpqiu
zpqiu requested a review from seonjinn October 13, 2026 08:06

@seonjinn seonjinn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the iteration on this one; the loss plumbing is in much better shape than the first revision.

Could you rebase onto main first? git merge-tree against origin/main (fa423c7) shows a single conflicting file, tests/unit/test_recipes_and_test_suites.py: main moved the nightly cap to 4830 while this branch takes it from 4815 to 4839. After the rebase the test name and assertion should read 4830 + 24 = 4854. Everything else merges cleanly, so the inline comments should survive the rebase.

TL;DR: no blocking correctness finding. 3 asks (config validation, ttt3-vs-ttt1 evidence plus a nightly acceptance gate, guard tests), 1 optional test ask, and 1 docs ask below. The two threads I opened earlier are fixed at this SHA.

What looks good

  • The LSE merge of chunked vocab-parallel logits in DraftTTTCrossEntropyLossFn reads correctly to me.
  • DraftLossStats as the single normalization seam, a sibling loss class instead of flags on DraftCrossEntropyLossFn, and the dense-reference tests in tests/unit/algorithms/test_draft_ttt_loss.py make the multi-pass path easy to reason about.
  • The split-path NotImplementedError and the Field(ge=1) / no-method cleanup are in.

Docs (not in the diff, so here)
docs/guides/eagle3-speculative-decoding.md still says sequence packing is supported (L67) with no caveat for ttt_steps > 1, and never mentions ttt_steps, ttt_pass_weights, the sequence_parallel=false requirement, or the flash-attn dependency for multi-pass. A short "Multi-pass TTT" subsection would close the unchecked docs box.

Context, no action

  • ttt_steps > 1 is read at about five sites (train.py forward and loss setup, wrapper.py, worker batch prep and split path, lm_policy.py guards). Fine for one Eagle-3 feature; once #3712/#3713 add drafters whose config class lacks ttt_steps, cfg["draft"].ttt_steps becomes an attribute error. The TODO(#3698) at wrapper.py:321 already points at the convergence.
  • examples/configs/ppo_math_1B_megatron_single_controller.yaml does not carry the new draft keys (defaults cover it); the grpo exemplar does.
  • "1 = stock single pass" holds as routing (the wrapper dispatches to DraftCrossEntropyLossFn), not as numeric identity of the TTT loss at d=1; worth phrasing that way in the description.

Not executed locally (no GPU / Linux-only lockfile); relying on your reported 32/60/6 passes and pyrefly-clean.

Comment thread nemo_rl/models/policy/draft_config.py Outdated
Comment on lines +55 to +65
@model_validator(mode="after")
def _validate_ttt_pass_weights(self) -> Self:
if (
self.ttt_pass_weights is not None
and len(self.ttt_pass_weights) != self.ttt_steps
):
raise ValueError(
f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, "
f"got {len(self.ttt_pass_weights)}."
)
return self

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 action item.

ttt_pass_weights is validated for length only, and the values feed the loss denominator directly (

denominator = (counts * self.weights).sum()
return (self.numerators * self.weights).sum() / (denominator + 1e-8)
). Three configs slip through today:

  • ttt_pass_weights: [0.0, 0.0, 0.0] trains with loss 0 / 1e-8 = 0; only the nightly min(draft_loss) > 0 check would notice.
  • A negative or NaN entry passes and surfaces as a negative or NaN train/draft_loss.
  • ttt_steps: 1 with ttt_pass_weights: [1.0] is accepted but silently ignored, since the weights only reach DraftTTTCrossEntropyLossFn on the multi-pass branch (
    self.multi_pass = ttt_steps > 1
    if self.multi_pass:
    self.draft_loss_fn: Any = DraftTTTCrossEntropyLossFn(
    vocab_parallel_group=vocab_parallel_group,
    pass_weights=ttt_pass_weights,
    ).

Would you extend the existing validator? Something like (needs import math at the top of the file):

Suggested change
@model_validator(mode="after")
def _validate_ttt_pass_weights(self) -> Self:
if (
self.ttt_pass_weights is not None
and len(self.ttt_pass_weights) != self.ttt_steps
):
raise ValueError(
f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, "
f"got {len(self.ttt_pass_weights)}."
)
return self
@model_validator(mode="after")
def _validate_ttt_pass_weights(self) -> Self:
if self.ttt_pass_weights is None:
return self
if len(self.ttt_pass_weights) != self.ttt_steps:
raise ValueError(
f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, "
f"got {len(self.ttt_pass_weights)}."
)
if any(not math.isfinite(w) or w < 0 for w in self.ttt_pass_weights):
raise ValueError("ttt_pass_weights entries must be finite and >= 0.")
if sum(self.ttt_pass_weights) <= 0:
raise ValueError("ttt_pass_weights must have a positive sum.")
if self.ttt_steps == 1:
raise ValueError(
"ttt_pass_weights is only used when ttt_steps > 1; drop it for "
"single-pass training."
)
return self

plus matching cases in tests/unit/models/policy/test_draft_config.py::test_eagle3_draft_config_validates_ttt_fields.

Action: reject non-finite, negative, all-zero, and ttt_steps == 1 weight lists, with tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 9c9b960: the weights are now typed list[Annotated[float, Field(ge=0, allow_inf_nan=False)]], and the validator also rejects an all-zero list and any weights with ttt_steps == 1, with cases added to test_eagle3_draft_config_validates_ttt_fields.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C2 · No action needed — resolved by removal. ttt_pass_weights is gone at this head, and ttt_step_loss_decay is validated instead (ge=0, finite) in beb5028. For later readers: the earlier reply here describes a typed-list validator that the rebase removed.

policy:
draft:
# One TTT pass per speculative token served by vLLM.
ttt_steps: 3

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 action item.

This recipe adds 24 GPU-hours to the nightly budget, and the PR motivates multi-pass TTT by serving-time acceptance, but neither the description nor the nightly checks carry an acceptance or throughput number yet (the follow-up mentioned in #3699 (comment) has not landed).

Two asks:

  1. A small table in the PR description for ttt_steps=3 vs ttt_steps=1 from the same init: s/step, peak train memory, vllm/spec_acceptance_length, vllm/spec_acceptance_rate-pos-{1,2,3}, generation tok/s. These are already logged (
    - vllm/spec_num_accepted_tokens: Total tokens accepted
    - vllm/spec_acceptance_length: Average accepted tokens per draft + 1
    - vllm/spec_acceptance_rate: Ratio of accepted to draft tokens
    - vllm/{metric}-{position}: Per-position acceptance counts
    - vllm/spec_acceptance_rate-pos-{position}: Per-position acceptance rates
    ), so no new instrumentation is needed. Also, the description's usage snippet shows loss_weight: 0.1 and a draft optimizer at lr: 5e-5, while this recipe inherits loss_weight: 1.0 and no draft optimizer from its parent. Which one was measured?
  2. The nightly asserts only draft_loss shape (
    'min(data["train/draft_loss"]) > 0' \
    'data["train/draft_loss"]["50"] < 4' \
    'mean(data["train/draft_loss_pass_3"], -5, -1) < 0.8 * data["train/draft_loss_pass_3"]["1"]'
    ), and the 0.8 * pass_3["1"] factor is not derived anywhere. A run where pass 3 learns nothing useful for serving can still pass. Since acceptance is logged, one line such as mean(data["vllm/spec_acceptance_length"], -5, -1) > <value from your run> would make this a regression test for what the feature is for.

Action: post ttt3 vs ttt1 numbers and add a vllm/spec_acceptance_length lower bound to the nightly checks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description now opens with ttt3-vs-ttt1 tables: an 8B random-init pair from the initial revision (acceptance length 1.65 → 1.93) and both 1.7B nightlies at this head, including peak train memory (11.19 vs 11.40 GiB). The usage snippet now matches what was measured (loss_weight: 1.0, no draft optimizer). 4a6e762 adds mean(spec_acceptance_length, -5, -1) > 2.05 to the nightly (step 1 is 1.90, runs end at 2.26–2.29), and the 0.8 factor leaves margin over the observed 0.59–0.60 pass-3 loss ratio.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C3 · No action needed — fixed in 42265a9. See the optional note on the 2.05 floor in the review body.

Comment thread nemo_rl/models/megatron/train.py Outdated
# multi-pass config would otherwise surface as a KeyError far
# from the cause.
multi_pass = "student_logits_by_pass" in data_dict
if multi_pass != (ttt_steps > 1):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 action item.

The two new guards have no tests: this config/forward consistency RuntimeError, and the split-path NotImplementedError (

if draft_enabled and self.cfg["draft"].ttt_steps > 1:
# DraftStepState, this path's deferred draft normalization, takes a
# single loss bin; the multi-pass loss has one bin per pass.
raise NotImplementedError(
"policy.draft.ttt_steps > 1 is not supported on the split "
"train-step path (single controller); use the batched train() "
"path."
)
). Both are CPU-testable in the style of tests/unit/models/policy/test_lm_policy_draft_guards.py::test_ttt_draft_with_sequence_packing_is_rejected (a stub cfg plus a data_dict with student_logits for this one; a stub worker with cfg["draft"].ttt_steps = 2 for the other).

Small question on the split-path guard: it fires inside the first train step, after model build and checkpoint load. Is there a reason it cannot sit next to the packing/SP checks in lm_policy.py, so a single-controller user fails at config time?

Action: add unit tests for the two guards; optionally hoist the split-path guard to config validation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6518b95 drops the config/forward check: the worker passes cfg["draft"].ttt_steps as draft_ttt_steps, so it compared the value with itself. The split-path guard now runs at the top of begin_train_step, tested in test_megatron_split_state.py. It can't sit in lm_policy.py because the policy config doesn't say which path runs: the single controller drives begin_train_step, while the data-plane GRPO trainer goes through train_presharded → batched train().

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C4 · No action needed — fixed in efa5683. The guard at the top of begin_train_step and its test cover this.

logits = logits.transpose(0, 1).contiguous()
return logits

def _ttt_rotary_pos_emb(self, ttt_pass: int, seq_len: int) -> Optional[Tensor]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 action item (optional, low priority).

forward_ttt is only exercised through a mock (tests/unit/models/megatron/test_train.py::test_forward_with_ttt_draft_model_uses_forward_ttt), and nothing covers _ttt_rotary_pos_emb. The pass-d offset (rotary_pos_emb(seq_len + d - 1)[d - 1:]) must match vLLM's cache position at depth d; an off-by-one would still train to a low loss and only show up as acceptance decay at positions 2-3. A table-driven test with a fake rotary_pos_emb returning arange, asserting the slice for d in {1, 2, 3}, is cheap and needs no flash-attn. A second small test that pass-2 output carries grad back to pass-1 inputs (no detach) would lock in the contract stated in the forward_ttt docstring.

Action: (optional) unit tests for the rotary offsets and the no-detach gradient chain.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the table-driven RoPE test in 4a6e762 (test_ttt_rotary_puts_pass_d_anchor_i_at_position_i_plus_d_minus_1, no GPU needed). I left out the gradient-chain test, since it would need flash-attn on a GPU and letting later passes backpropagate into earlier ones is the standard EAGLE-3 TTT setup, which SpecForge trains the same way.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C5 · No action needed — RoPE offset test added in 42265a9. OK to skip the grad-chain test (GPU-only).

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 16, 2026
@zpqiu
zpqiu requested a review from seonjinn October 22, 2026 05:36
MarkXCloud and others added 5 commits November 2, 2026 01:58
Extends the single-pass EAGLE-3 draft co-training path on Megatron to
multi-pass TTT (policy.draft.ttt_steps > 1):

- EagleModel.forward_ttt: sequential draft passes with the two-part
  trunk+branch flash attention (exact joint-LSE merge) folded into
  eagle.py, inference-aligned per-pass RoPE offsets, non-detached
  hidden-state chain between passes.
- DraftTTTCrossEntropyLossFn: the multi-pass sibling of the single-pass
  DraftCrossEntropyLossFn (which stays the ttt_steps == 1 loss,
  including the packed layout). Per-pass slicing convention (pass d
  predicts x_{i+d+1}); each pass is one DraftLossStats bin from the
  streaming vocab-parallel soft-CE, weighted by
  ttt_step_loss_decay**(d-1) and normalized by per-pass global valid
  counts threaded from the worker (DP-only all-reduce). The d2t teacher
  gather is tiled by policy.draft.token_chunk_size.
- The Megatron path now reads Eagle3DraftConfig's ttt_steps and
  ttt_step_loss_decay, shared with the Automodel TTT path (default
  ttt_steps=3; ttt_steps >= 1 is validated on the config). On Megatron,
  ttt_steps > 1 rejects sequence packing (until NVIDIA-NeMo#4059), sequence
  parallelism, and the split train-step path (its DraftStepState takes a
  single loss bin). The single-pass eagle3 recipe pins ttt_steps: 1.
- Aux-layer ids resolved rank-independently
  (resolve_draft_aux_layer_ids) and threaded into the hidden-state
  capture, so PP stages cannot disagree about the P2P schedule.
- vLLM: un-share the drafter lm_head before loading a trained head
  (refit would silently overwrite the target's head through vLLM's
  share-on-identical-weights alias), preserving vLLM's weight-loader
  attrs on the private copy.
- Nightly grpo-qwen3-1.7b-1n8g-megatron-eagle3-ttt3: the eagle3
  nightly with ttt_steps=3, checking that the last pass learns (nightly
  GPU-hour cap 5203 -> 5227).

Signed-off-by: Zili Wang <1348131731@qq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject negative and non-finite decay factors: a negative factor flips
the sign of every other pass's weight, and both enter the loss
denominator directly.

Signed-off-by: Zili Wang <1348131731@qq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…step

Move the ttt_steps > 1 rejection from the first microbatch to the start of
the split train step, with a test. Drop the forward/config consistency
check in the loss setup: the only caller passes cfg ttt_steps as
draft_ttt_steps, so it compared the value with itself.

Signed-off-by: Zili Wang <1348131731@qq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…acceptance

Assert pass d places anchor i at position i + d - 1, vLLM's position for
the d-th speculative token. The ttt3 nightly now also requires the last-5
acceptance length above 2.05 (step 1: 1.90; runs end at 2.26-2.29).

Signed-off-by: Zili Wang <1348131731@qq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a Multi-pass TTT section and the ttt_steps / ttt_step_loss_decay
knobs, note that ttt_steps defaults to 3 so single-pass configs set it
explicitly, and scope the sequence-packing support note to
ttt_steps == 1.

Signed-off-by: Zili Wang <1348131731@qq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MarkXCloud
MarkXCloud marked this pull request as ready for review November 3, 2026 02:02
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Nov 3, 2026

@seonjinn seonjinn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of round 1. Thanks for the fixes. The split-path guard at begin_train_step, the RoPE offset test and the nightly gates address the round-1 asks.

  1. (C1) The draft lm_head swap happens after CUDA graph capture, so the draft decode graphs may keep reading the target head.

Optional notes with no inline comment. Confidence is below the staging bar, so this is the author's call:

  • resolve_draft_aux_layer_ids has no unit test. Its three branches (checkpoint ids, config ids, modelopt default) can run on CPU with a patched AutoConfig.
  • The ttt3 nightly acceptance floor > 2.05 (line 52) is below the ttt1 acceptance quoted in the thread (2.24). Only the pass-3 loss ratio check separates ttt3 from ttt1. The floor would also not catch C1.
  • Pass chaining in forward_ttt is covered only by @requires_flash_attn GPU tests and the nightly. Which CI lane runs those?
  • The chunked TP gather and d2t path in loss/utils.py has no numeric test against the unchunked path.
  • Megatron now honors the existing default ttt_steps: 3. Megatron eagle3 users with sequence_parallel or packing get a startup ValueError until they set ttt_steps: 1. This is worth one line in the release note.
  • train/draft_loss for ttt1 and train/draft_loss_pass_1 for ttt>1 use different target offsets (documented in loss_functions.py), so curves across the two are not directly comparable.

Not executed locally (no GPU, Linux-only lockfile).

Merge note: this branch conflicts with latest main in nemo_rl/algorithms/loss/wrapper.py, tests/unit/models/policy/test_draft_config.py and tests/unit/test_recipes_and_test_suites.py.

Generated by Claude Code

private_lm_head.parameters(), target_lm_head.parameters()
):
private_param.__dict__.update(source_param.__dict__)
draft_model.lm_head = private_lm_head

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C1 · 1 action item. TL;DR — draft decode CUDA graphs are captured at warm-up and keep the target head's address, so the new draft head that _unshare_draft_lm_head swaps in at refit is never read by them.

_unshare_draft_lm_head is called only from _load_draft_weights, which runs only on refit. That is after engine init and after CUDA graph capture. The swap rebinds draft_model.lm_head to a deep copy at a new address. It does not copy into the old storage. Graphs that were captured earlier still point at the target head.

The target head stays correct, so the PR's goal for the target model holds. The trained draft head goes to storage that the draft decode graphs never read. Acceptance stays above zero, so no check fails and the problem stays hidden. The default config has enforce_eager set to false, so graphs are on by default.

The docstring names SpecDecodeBaseProposer._maybe_share_lm_head, which is not the default path. The default V2 path aliases the head in load_eagle_model.

AI-1

Move the un-share to engine load, before CUDA graph capture, when draft co-training streams an lm_head. Capture then records the private head's address. Also change the docstring at line 1153 to name load_eagle_model (vLLM eagle/utils.py) as the default path. Please post the engine-log lines that show the runner and cudagraph mode from the ttt3 nightly. This is not a suggestion block because the fix moves code to engine init, outside this anchor.

vLLM v0.29.0 call chain behind the claim (not verified, no GPU environment)
  • In the V2 runner speculator, draft decodes at positions above 0 use FULL_DECODE_ONLY graphs when the decode mode is FULL: speculator.py#L136-L138.
  • Capture records the model forward, compute_logits and sample as one FULL graph: speculator.py#L157-L161.
  • Eagle3 compute_logits reads self.lm_head: llama_eagle3.py#L339.
  • vLLM aliases the draft head to the target head when the draft has no has_own_lm_head flag: eagle/utils.py#L12-L16 and eagle/utils.py#L83-L90. The flag is always missing under load_format=dummy, which is how NeMo-RL starts the engine before refit.
  • Draft prefill (position 0) graph mode was not confirmed.
  • Not verified - no GPU environment.

Comment thread nemo_rl/models/policy/draft_config.py Outdated
Comment on lines +55 to +65
@model_validator(mode="after")
def _validate_ttt_pass_weights(self) -> Self:
if (
self.ttt_pass_weights is not None
and len(self.ttt_pass_weights) != self.ttt_steps
):
raise ValueError(
f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, "
f"got {len(self.ttt_pass_weights)}."
)
return self

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C2 · No action needed — resolved by removal. ttt_pass_weights is gone at this head, and ttt_step_loss_decay is validated instead (ge=0, finite) in beb5028. For later readers: the earlier reply here describes a typed-list validator that the rebase removed.

policy:
draft:
# One TTT pass per speculative token served by vLLM.
ttt_steps: 3

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C3 · No action needed — fixed in 42265a9. See the optional note on the 2.05 floor in the review body.

Comment thread nemo_rl/models/megatron/train.py Outdated
# multi-pass config would otherwise surface as a KeyError far
# from the cause.
multi_pass = "student_logits_by_pass" in data_dict
if multi_pass != (ttt_steps > 1):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C4 · No action needed — fixed in efa5683. The guard at the top of begin_train_step and its test cover this.

logits = logits.transpose(0, 1).contiguous()
return logits

def _ttt_rotary_pos_emb(self, ttt_pass: int, seq_len: int) -> Optional[Tensor]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C5 · No action needed — RoPE offset test added in 42265a9. OK to skip the grad-chain test (GPU-only).

@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) community-request Documentation Improvements or additions to documentation waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants