Repository navigation
feat(spec-decode): multi-pass TTT training for EAGLE-3 drafts (Megatron) - #3699
MarkXCloud wants to merge 5 commits into
Conversation
ef0fe42 to
e93d5d8
Compare
zpqiu
left a comment
There was a problem hiding this comment.
Review of the multi-pass TTT draft-training path, focused on correctness and CI state. This was a deep pass — six reviewers across the RL code, Megatron/flash-attn call chains, the vLLM refit path, and the test suite, with findings adversarially challenged before landing here. Roughly 35 raw findings were reduced to the 12 comments below; a dozen were dropped as unreachable, pre-existing, or style, and nine were withdrawn by their own authors under challenge.
The hardest part of this PR is correct. The trunk+branch joint-LSE attention was verified three independent ways: analytically against pinned flash-attn 2.8.1 source (the softmax_lse natural-log base and the FA-backward-with-joint-(out, lse) trick both check out), empirically on an H100 against a dense fp32 full-attention reference (fwd 3.09e-03, worst gradient 5.44e-03 in bf16), and by your own tests, which compare against modelopt's set_multi_step_attention_mask as an external oracle. ChunkedDistributedCrossEntropy was separately confirmed correct on two GPUs against a full-vocab reference. The per-pass RoPE offsets, GQA head grouping, and DP-only count all-reduce all check out too. That's a stronger evidence base than most PRs this size carry, and the AIME24 three-arm plot is a genuinely useful convergence check.
The main ask is CI. Seven tests currently fail and pyrefly reports a new error. Since pyproject.toml:723 puts -x in addopts, a local run stops at the first one — --maxfail=100 shows them all, which may be why this wasn't visible. Three of the failures are simply the Megatron-LM submodule moving under you (main bumped it in a0edb27 and upstream fixed the underlying issue), so the fix there is deleting code rather than writing it.
Two items are worth a look even though they aren't reachable on the shipped recipes — the vLLM drafter lookup and the PP aux-layer divergence — because both fail silently or fatally rather than loudly, and both have small fixes.
Documentation comments are deliberately omitted, since that box is unchecked. One thing that would round out the evidence when you're ready for review: spec-decode acceptance rate (mean accepted tokens/step) for ttt_steps=3 vs the pretrained drafter, since that's the headline metric for the rollout speedup the perf plot is showing.
Generated by Claude Code
| return | ||
| import copy | ||
|
|
||
| draft_model.lm_head = copy.deepcopy(target_lm_head) |
There was a problem hiding this comment.
nemo_rl/models/generation/vllm/vllm_backend.py:639
pyrefly fails on this line, and while you're here there's a subtlety worth guarding.
The typecheck (a pre-commit hook, pass_filenames: false) reports a new error introduced by this line — main has 0, this branch has 1:
ERROR Argument `Any | None` is not assignable to parameter `value` with type `Module | Tensor`
in function `torch.nn.modules.module.Module.__setattr__` [bad-argument-type]
--> nemo_rl/models/generation/vllm/vllm_backend.py:639
An isinstance(target_lm_head, torch.nn.Module) narrow above the storage check fixes it.
The separate subtlety: torch.nn.Parameter.__deepcopy__ rebuilds a bare Parameter, dropping attributes vLLM attaches via set_weight_attrs (vocab_parallel_embedding.py#L58). Reproduced on torch 2.11.0+cu130:
ORIGINAL attrs: ['weight_loader', 'input_dim', 'output_dim']
DEEPCOPY attrs: []
load_weights resolves the loader off the param (models/utils.py#L277) and would fall back to default_weight_loader, which can't shard and asserts under TP>1. The deep-copied head also carries the target's org_vocab_size, which _trim_vocab_padding then reads for the draft.
Worth noting this is currently unreachable, so it's a latent issue rather than something you're hitting: NeMo-RL's own has_own_lm_head patch (nemo_rl/models/generation/vllm/patches.py:161) sets draft_vocab_size != vocab_size, and both shipped drafters are truncated-vocab (AngelSlim/Qwen3-{1.7B,8B}_eagle3: vocab_size=151936, draft_vocab_size=32000), so vLLM never shares the head and this method early-returns. It would bite a full-vocab drafter. Building a fresh module and copy_()-ing the data is more robust than deepcopy-plus-reattach.
There was a problem hiding this comment.
Added the isinstance narrow (the file is pyrefly-clean now). For the latent attrs issue I kept deepcopy but re-bind the source params' __dict__ onto the copies, with a test pinning weight_loader survival — reattach avoids reproducing ParallelLMHead's ctor args.
| if draft_model is not None: | ||
| configured_aux_layers = getattr( | ||
| getattr(draft_model, "config", None), | ||
| "eagle_aux_hidden_state_layer_ids", |
There was a problem hiding this comment.
nemo_rl/models/megatron/train.py:295
Under PP > 1 this resolves to a different aux-layer list on each stage.
draft_model is attached only to the post-process chunk, so find_draft_owner_chunk returns None on every non-last stage (nemo_rl/models/megatron/draft/utils.py:975) and aux_layer_indices is None there. HiddenStateCapture.__init__ then falls back to get_eagle3_aux_hidden_state_layers(num_layers) (nemo_rl/models/megatron/draft/hidden_capture.py:66), while the last stage uses the configured ids. _gather_distributed iterates that list posting one send/recv per index (nemo_rl/models/megatron/draft/hidden_capture.py:251), so two different lists means mismatched counts — a wrong-tensor match that shapes won't catch, then a desync.
This is not reachable on the shipped recipes: neither AngelSlim/Qwen3-1.7B_eagle3 nor Qwen3-8B_eagle3 carries eagle_aux_hidden_state_layer_ids, so build_draft_model falls back to the same get_eagle3_aux_hidden_state_layers(model_provider.num_layers) and the lists coincide. The reachable trigger is the documented knob: policy.draft.aux_layer_indices: [...] with pipeline_model_parallel_size > 1 and packing off (which the guard at nemo_rl/models/policy/lm_policy.py:146 permits).
Flagging honestly: this is a static trace — I did not run it on two PP stages. Resolving the indices from a rank-independent source (or all-gathering them over the PP group and asserting agreement) would remove the divergence.
Minor, same block: the nested getattr(getattr(draft_model, "config", None), ...) can be a direct read since build_draft_model always sets that attribute.
There was a problem hiding this comment.
Fixed at the root — new resolve_draft_aux_layer_ids() resolves the list rank-independently and is threaded into both forward paths, and build_draft_model uses the same resolver so the draft and the capture can't disagree. The nested getattr is a direct read now.
| import pytest | ||
| import torch | ||
|
|
||
| pytest.importorskip("vllm", reason="vllm_backend requires a vLLM install") |
There was a problem hiding this comment.
tests/unit/models/generation/test_vllm_draft_lm_head_unshare.py:30
This file never executes in CI — verified Running 0 items in this shard.
L0_Unit_Tests_Vllm_1.sh runs two passes: a base run that globs test_vllm*.py with --no-sync (vLLM is an optional extra, so importorskip skips the module), and a --extra vllm --vllm-only catch-all that collects only marker-carrying items. With importorskip but no marker, the file falls through both. test_nccl_reshard_backend.py is the exemplar — it pairs the marker with an importorskip, exactly this situation.
Verified fix takes it from 0 collected to 3 passing:
| pytest.importorskip("vllm", reason="vllm_backend requires a vLLM install") | |
| pytestmark = pytest.mark.vllm | |
| pytest.importorskip("vllm", reason="vllm_backend requires a vLLM install") |
Worth pairing with one assertion, since the current stubs would pass even with the weight_loader issue above present (draft_model is a SimpleNamespace, so nn.Module.__setattr__ is never exercised, and no ParallelLMHead is ever built): after un-sharing, assert hasattr(draft_model.lm_head.weight, "weight_loader"). The _load_draft_weights raise path is also untested — a pytest.raises there would have caught the PP case above.
There was a problem hiding this comment.
Took the fix (marker + importorskip), added the weight_loader assertion, and added the _load_draft_weights raise-path tests you suggested.
seonjinn
left a comment
There was a problem hiding this comment.
Review
Read this branch merged with current main (clean merge, 59 files). @zpqiu has already covered the correctness of the joint-LSE trunk+branch attention and the CI state, so this review does not revisit either — everything below is either a config-plumbing defect or a collision with the drafter stack that is landing in parallel.
One framing note before the findings: CI on this PR runs only DCO, the submodule comment, semantic-pull-request and triage. There is no unit-test lane, so the seven failures reported from a local run will not surface here no matter how many times it is re-run, and none of the defects below are catchable by the checks currently on the PR.
Actionable
1. Multi-pass normalization is silently wrong on the split train-step path. megatron_policy_worker.py:1580 threads draft_ttt_steps=draft_ttt_steps into the split step but not global_draft_pass_counts, while the monolithic path passes both (:978 and :983). With ttt_steps > 1 on the split path the loss therefore receives global_draft_pass_counts=None and falls back to denominator = global_valid_toks (loss_functions.py:218-219) — the policy's token count, not the intended sum_d alpha_d * count_d. The draft gradient is scaled by the wrong denominator, by roughly the ratio of the two counts, with no error and no metric that would look obviously wrong.
Either compute the counts on the split path as well, or reject the combination outright. I would reject for now: raise NotImplementedError when ttt_steps > 1 and the split path is selected. That also becomes mandatory once #3706 lands — see item 5 of the follow-up comment.
2. ttt_steps: 0 is silently coerced to 1. int(draft_cfg.get("ttt_steps", 1) or 1) (train.py:659, and the same expression at megatron_policy_worker.py:872 and :1547) turns 0 into 1 because 0 is falsy, while -1 passes straight through to whatever fails first. A user who writes ttt_steps: 0 meaning "off" gets a single-pass run and no message. This wants one validated field rather than three copies of a defensive coercion — see item 2 of the follow-up comment.
3. The weight-decay-only draft optimizer config is rejected. DraftOptimizerConfigOverrideProvider.build_config_overrides (draft/utils.py:1328-1334) raises ValueError: [draft] draft (lr, min_lr) must differ from the policy's. whenever the effective (max_lr, min_lr) equals the policy's. Setting only draft.weight_decay leaves both LRs unset, so effective_max_lr/effective_min_lr fall back to the policy's values and the raise fires — even though the requested override (start_wd/end_wd) is perfectly distinct and would work. Giving the drafter a different weight decay at the same LR is a reasonable thing to want and is currently unreachable.
The raise exists because the group's resume identity was assumed to be just (max_lr, min_lr). It is not, any more — see item 1 of the follow-up comment, which is also why I would delete this code path rather than patch the condition.
4. ttt_pass_weights length is validated per microbatch, not at startup. The check lives in DraftTTTCrossEntropyLossFn.__call__ (loss_functions.py:191-195), so a mismatched list is caught on the first forward of the first microbatch rather than at config parse. The length is fully known at config time. Same for the global_draft_pass_counts shape check right below it, which is derived from the same ttt_steps.
5. The forward-side pass count and the loss-side pass count come from different places. The forward selects forward_ttt from the draft_ttt_steps function parameter, which defaults to 1 (train.py:233, used at :369), while DraftLossWrapper is constructed with ttt_steps read from self.cfg["draft"] (train.py:659, :683). Both call sites in this PR pass the kwarg, so the two agree today. But the default makes the divergence silent-by-construction for the next caller: omit the kwarg with ttt_steps: 3 in the config and the forward produces single-pass output while the loss asks for student_logits_by_pass, i.e. a KeyError some distance from the cause. Deriving both from one config object removes the trap.
6. raise ValueError(f"Unknown draft_method ...") is unreachable. method is read as draft_cfg.get("method", "eagle3") (train.py:658) but is declared in no schema and set in no YAML in the repo, so the only reachable value is the default. This is a symptom of the selector duplication rather than a bug on its own — see item 4 of the follow-up comment.
Nit
self.seq_chunk_size defaults to 4096 in DraftTTTCrossEntropyLossFn.__init__ (loss_functions.py:155), but train.py:668-671 only forwards loss_seq_chunk_size when ttt_steps > 1. In the single-pass case the key is accepted by the config and then ignored. Harmless today because DraftCrossEntropyLossFn does not chunk, but it reads as a knob that works everywhere.
Verified clean
The guard placement is right: ttt_steps > 1 with sequence_packing is rejected early at lm_policy.py:176-188, where the user sees it at setup, and CP / sequence-parallel for ttt_steps > 1 are rejected in build_draft_model (draft/utils.py:1471-1481). The per-pass metric denominators are correctly the global per-pass counts rather than per-microbatch means (loss_functions.py:247-259) — the comment there explaining the ~512x inflation at gbs 512 is worth keeping.
ChunkedDistributedCrossEntropy's memory argument holds: saving the bf16 logits references and recomputing the chunk softmaxes in backward is the right trade when several TTT passes stack before a shared backward, and it mirrors ChunkedDistributedLogprob. My only objection to it is that #3703 already ships the same kernel — see the follow-up comment.
Method note
Nothing except lint was executed: torch is not importable in the review environment, so all of the above is read from source on the merged tree. ruff 0.9.9 check and format --check are both clean on the merged tree (640 files formatted), so the reported CI failures are test failures, not lint. The submodule pin was resolved through 3rdparty/Megatron-Bridge-workspace/Megatron-Bridge at b11414c71b15e54d333eb49346ed199f20fa9021.
Integration suggestions for the DFlash/DSpark drafter stackThanks again for this work. Would it be okay if I leave a few integration-focused suggestions here? I am happy for your team to continue owning the EAGLE-3 TTT implementation, and I do not intend to propose changes to the core algorithm — the TTT loss, the joint-LSE trunk+branch attention, and the multi-pass schedule are yours. What I would like to align on is the surrounding plumbing: the typed draft config, the loss-statistics interface, and the optimizer/refit structures, so that this integrates cleanly with the DFlash/DSpark online-training PRs landing alongside it ( Concretely, the four conventions I would suggest converging on:
On timing, since none of the stack has landed yet. I am not asking you to depend on unmerged branches, and this PR should not be held waiting on them. The point of raising it now is that both sides can still agree cheaply on a shared shape, rather than discovering it as a conflict at merge time. The ordinary rule applies: whichever lands first stays, the second rebases. The items below are written for the case where the stack lands first, since that is the current plan of record — if it slips and this PR lands first, the same items apply in reverse and I will carry them on the stack side instead. Either way, most of the mechanical work below is work I am happy to do rather than hand to you; tell me which parts you would rather own. In practice, only item 4 is worth doing before the stack merges. The rest is agreement-in-principle. I have written these as concrete diffs rather than a neutral comparison, because a comparison leaves the decision unmade — not because the choice is settled. If you disagree on any of them, that is exactly the discussion worth having before either side merges. Everything cited was read from this branch merged with current 1. Optimizer overrides: converge on #3707's structureBoth PRs add a function with the same name, wired into the same call:
Neither is in Suggestion: use #3707's provider instead of the one here. That means removing, from
and the wiring at Three reasons the stack's version is the one to keep, beyond merge order:
One thing to carry over rather than drop. The two disagree on what
If the constant-LR default is what your AIME24 runs used, that is worth saying on #3707 before it merges, because rebasing onto #3707 silently changes it. Both behaviours stay reachable either way by setting 2. EAGLE fields: declare them on
|
seonjinn
left a comment
There was a problem hiding this comment.
Two of the items from my earlier comments as applicable suggestions, so they are easier to take or reject on the spot. Both are standalone — neither depends on the drafter stack landing.
| draft_ttt_steps = ( | ||
| int(self.cfg["draft"].get("ttt_steps", 1) or 1) if draft_enabled else 1 | ||
| ) |
There was a problem hiding this comment.
Finding 1: this path threads draft_ttt_steps into megatron_forward_backward (:1578) but never computes global_draft_pass_counts, unlike the monolithic path (:881, passed at :976). The draft loss then falls back to denominator = global_valid_toks — the policy token count rather than sum_d alpha_d * count_d — so the draft gradient is mis-scaled with no error and no metric that looks wrong.
Failing fast is the smaller change; computing the counts here would also be correct if you prefer that.
| draft_ttt_steps = ( | |
| int(self.cfg["draft"].get("ttt_steps", 1) or 1) if draft_enabled else 1 | |
| ) | |
| draft_ttt_steps = ( | |
| int(self.cfg["draft"].get("ttt_steps", 1) or 1) if draft_enabled else 1 | |
| ) | |
| if draft_ttt_steps > 1: | |
| raise NotImplementedError( | |
| "policy.draft.ttt_steps > 1 is not supported on the split train-step " | |
| "path: global_draft_pass_counts is not computed here, so the draft " | |
| "loss would normalize by the policy token count instead of " | |
| "sum_d alpha_d * count_d." | |
| ) |
There was a problem hiding this comment.
Adopted: the split train-step path now raises NotImplementedError for ttt_steps > 1, which #3706's single-bin DraftStepState requires anyway.
| draft_cfg = self.cfg["draft"] | ||
| draft_method = draft_cfg.get("method", "eagle3") | ||
| ttt_steps = int(draft_cfg.get("ttt_steps", 1) or 1) | ||
| # Ctor kwargs for the method's draft LossFn (uniform shape; | ||
| # DraftLossWrapper splats them into the selected class). Only | ||
| # the multi-pass class takes pass weights. | ||
| draft_loss_kwargs: dict[str, Any] = ( | ||
| {"pass_weights": draft_cfg.get("ttt_pass_weights")} | ||
| if ttt_steps > 1 | ||
| else {} | ||
| ) | ||
| if ttt_steps > 1 and draft_cfg.get("loss_seq_chunk_size") is not None: | ||
| draft_loss_kwargs["seq_chunk_size"] = int( | ||
| draft_cfg["loss_seq_chunk_size"] | ||
| ) | ||
| loss_fn_wrapped = DraftLossWrapper( | ||
| loss_fn=loss_fn_wrapped, | ||
| prepare_fn=prepare_loss_input_wrapped, | ||
| data_dict=data_dict, | ||
| loss_weight=float(self.cfg["draft"]["loss_weight"]), | ||
| loss_weight=float(draft_cfg["loss_weight"]), | ||
| draft_loss_kwargs=draft_loss_kwargs, | ||
| global_draft_pass_counts=global_draft_pass_counts, | ||
| vocab_parallel_rank=get_tensor_model_parallel_rank(), | ||
| vocab_parallel_group=get_tensor_model_parallel_group(), | ||
| context_parallel_group=get_context_parallel_group(), | ||
| draft_method=draft_method, | ||
| ttt_steps=ttt_steps, |
There was a problem hiding this comment.
Two things folded into one suggestion, both from the earlier comments:
ttt_steps: 0is silently coerced to 1.int(... or 1)turns0into1because0is falsy, while-1passes through to whatever fails first. Explicit validation instead.- The
methodselector.draft_cfg.get("method", "eagle3")is declared in no schema and set in no YAML, so"eagle3"is the only reachable value and theUnknown draft_methodraise inDraftLossWrapperis dead.draft_methodalready defaults to"eagle3"in the wrapper signature (wrapper.py:268), so dropping the read here is behaviour-preserving today — and it avoids a second selector disagreeing withspeculator_typeonce the DFlash/DSpark configs land.
Happy to drop the second half if you would rather keep method for now; the first half stands on its own.
| draft_cfg = self.cfg["draft"] | |
| draft_method = draft_cfg.get("method", "eagle3") | |
| ttt_steps = int(draft_cfg.get("ttt_steps", 1) or 1) | |
| # Ctor kwargs for the method's draft LossFn (uniform shape; | |
| # DraftLossWrapper splats them into the selected class). Only | |
| # the multi-pass class takes pass weights. | |
| draft_loss_kwargs: dict[str, Any] = ( | |
| {"pass_weights": draft_cfg.get("ttt_pass_weights")} | |
| if ttt_steps > 1 | |
| else {} | |
| ) | |
| if ttt_steps > 1 and draft_cfg.get("loss_seq_chunk_size") is not None: | |
| draft_loss_kwargs["seq_chunk_size"] = int( | |
| draft_cfg["loss_seq_chunk_size"] | |
| ) | |
| loss_fn_wrapped = DraftLossWrapper( | |
| loss_fn=loss_fn_wrapped, | |
| prepare_fn=prepare_loss_input_wrapped, | |
| data_dict=data_dict, | |
| loss_weight=float(self.cfg["draft"]["loss_weight"]), | |
| loss_weight=float(draft_cfg["loss_weight"]), | |
| draft_loss_kwargs=draft_loss_kwargs, | |
| global_draft_pass_counts=global_draft_pass_counts, | |
| vocab_parallel_rank=get_tensor_model_parallel_rank(), | |
| vocab_parallel_group=get_tensor_model_parallel_group(), | |
| context_parallel_group=get_context_parallel_group(), | |
| draft_method=draft_method, | |
| ttt_steps=ttt_steps, | |
| draft_cfg = self.cfg["draft"] | |
| raw_ttt_steps = draft_cfg.get("ttt_steps", 1) | |
| ttt_steps = 1 if raw_ttt_steps is None else int(raw_ttt_steps) | |
| if ttt_steps < 1: | |
| raise ValueError( | |
| f"policy.draft.ttt_steps must be >= 1, got {ttt_steps}" | |
| ) | |
| # Ctor kwargs for the draft LossFn (uniform shape; DraftLossWrapper | |
| # splats them into the selected class). Only the multi-pass class | |
| # takes pass weights. | |
| draft_loss_kwargs: dict[str, Any] = ( | |
| {"pass_weights": draft_cfg.get("ttt_pass_weights")} | |
| if ttt_steps > 1 | |
| else {} | |
| ) | |
| if ttt_steps > 1 and draft_cfg.get("loss_seq_chunk_size") is not None: | |
| draft_loss_kwargs["seq_chunk_size"] = int( | |
| draft_cfg["loss_seq_chunk_size"] | |
| ) | |
| loss_fn_wrapped = DraftLossWrapper( | |
| loss_fn=loss_fn_wrapped, | |
| prepare_fn=prepare_loss_input_wrapped, | |
| data_dict=data_dict, | |
| loss_weight=float(draft_cfg["loss_weight"]), | |
| draft_loss_kwargs=draft_loss_kwargs, | |
| global_draft_pass_counts=global_draft_pass_counts, | |
| vocab_parallel_rank=get_tensor_model_parallel_rank(), | |
| vocab_parallel_group=get_tensor_model_parallel_group(), | |
| context_parallel_group=get_context_parallel_group(), | |
| ttt_steps=ttt_steps, | |
| ) |
There was a problem hiding this comment.
Both done: ttt_steps is Annotated[int, Field(ge=1)] on Eagle3DraftConfig, and the method read and draft_method parameter are gone, so speculator_type is the only selector and ttt_steps > 1 picks the multi-pass loss.
|
@zpqiu Thanks for the review — all seven test failures and the pyrefly error are fixed in the updated commit; replied on each comment in-thread. Test evidence (container run against the pinned submodules, Acceptance-rate data ( Could a maintainer post |
|
/ok to test 5ab691c |
5e503aa to
abfd277
Compare
seonjinn
left a comment
There was a problem hiding this comment.
Thanks for the iteration on this one; the loss plumbing is in much better shape than the first revision.
Could you rebase onto main first? git merge-tree against origin/main (fa423c7) shows a single conflicting file, tests/unit/test_recipes_and_test_suites.py: main moved the nightly cap to 4830 while this branch takes it from 4815 to 4839. After the rebase the test name and assertion should read 4830 + 24 = 4854. Everything else merges cleanly, so the inline comments should survive the rebase.
TL;DR: no blocking correctness finding. 3 asks (config validation, ttt3-vs-ttt1 evidence plus a nightly acceptance gate, guard tests), 1 optional test ask, and 1 docs ask below. The two threads I opened earlier are fixed at this SHA.
What looks good
- The LSE merge of chunked vocab-parallel logits in
DraftTTTCrossEntropyLossFnreads correctly to me. DraftLossStatsas the single normalization seam, a sibling loss class instead of flags onDraftCrossEntropyLossFn, and the dense-reference tests intests/unit/algorithms/test_draft_ttt_loss.pymake the multi-pass path easy to reason about.- The split-path
NotImplementedErrorand theField(ge=1)/ no-methodcleanup are in.
Docs (not in the diff, so here)
docs/guides/eagle3-speculative-decoding.md still says sequence packing is supported (L67) with no caveat for ttt_steps > 1, and never mentions ttt_steps, ttt_pass_weights, the sequence_parallel=false requirement, or the flash-attn dependency for multi-pass. A short "Multi-pass TTT" subsection would close the unchecked docs box.
Context, no action
ttt_steps > 1is read at about five sites (train.pyforward and loss setup,wrapper.py, worker batch prep and split path,lm_policy.pyguards). Fine for one Eagle-3 feature; once #3712/#3713 add drafters whose config class lacksttt_steps,cfg["draft"].ttt_stepsbecomes an attribute error. TheTODO(#3698)atwrapper.py:321already points at the convergence.examples/configs/ppo_math_1B_megatron_single_controller.yamldoes not carry the new draft keys (defaults cover it); the grpo exemplar does.- "1 = stock single pass" holds as routing (the wrapper dispatches to
DraftCrossEntropyLossFn), not as numeric identity of the TTT loss at d=1; worth phrasing that way in the description.
Not executed locally (no GPU / Linux-only lockfile); relying on your reported 32/60/6 passes and pyrefly-clean.
| @model_validator(mode="after") | ||
| def _validate_ttt_pass_weights(self) -> Self: | ||
| if ( | ||
| self.ttt_pass_weights is not None | ||
| and len(self.ttt_pass_weights) != self.ttt_steps | ||
| ): | ||
| raise ValueError( | ||
| f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, " | ||
| f"got {len(self.ttt_pass_weights)}." | ||
| ) | ||
| return self |
There was a problem hiding this comment.
1 action item.
ttt_pass_weights is validated for length only, and the values feed the loss denominator directly (
RL/nemo_rl/algorithms/loss/draft.py
Lines 86 to 87 in abfd277
ttt_pass_weights: [0.0, 0.0, 0.0]trains with loss0 / 1e-8 = 0; only the nightlymin(draft_loss) > 0check would notice.- A negative or NaN entry passes and surfaces as a negative or NaN
train/draft_loss. ttt_steps: 1withttt_pass_weights: [1.0]is accepted but silently ignored, since the weights only reachDraftTTTCrossEntropyLossFnon the multi-pass branch ().RL/nemo_rl/algorithms/loss/wrapper.py
Lines 324 to 328 in abfd277
Would you extend the existing validator? Something like (needs import math at the top of the file):
| @model_validator(mode="after") | |
| def _validate_ttt_pass_weights(self) -> Self: | |
| if ( | |
| self.ttt_pass_weights is not None | |
| and len(self.ttt_pass_weights) != self.ttt_steps | |
| ): | |
| raise ValueError( | |
| f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, " | |
| f"got {len(self.ttt_pass_weights)}." | |
| ) | |
| return self | |
| @model_validator(mode="after") | |
| def _validate_ttt_pass_weights(self) -> Self: | |
| if self.ttt_pass_weights is None: | |
| return self | |
| if len(self.ttt_pass_weights) != self.ttt_steps: | |
| raise ValueError( | |
| f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, " | |
| f"got {len(self.ttt_pass_weights)}." | |
| ) | |
| if any(not math.isfinite(w) or w < 0 for w in self.ttt_pass_weights): | |
| raise ValueError("ttt_pass_weights entries must be finite and >= 0.") | |
| if sum(self.ttt_pass_weights) <= 0: | |
| raise ValueError("ttt_pass_weights must have a positive sum.") | |
| if self.ttt_steps == 1: | |
| raise ValueError( | |
| "ttt_pass_weights is only used when ttt_steps > 1; drop it for " | |
| "single-pass training." | |
| ) | |
| return self |
plus matching cases in tests/unit/models/policy/test_draft_config.py::test_eagle3_draft_config_validates_ttt_fields.
Action: reject non-finite, negative, all-zero, and ttt_steps == 1 weight lists, with tests.
There was a problem hiding this comment.
Done in 9c9b960: the weights are now typed list[Annotated[float, Field(ge=0, allow_inf_nan=False)]], and the validator also rejects an all-zero list and any weights with ttt_steps == 1, with cases added to test_eagle3_draft_config_validates_ttt_fields.
There was a problem hiding this comment.
C2 · No action needed — resolved by removal. ttt_pass_weights is gone at this head, and ttt_step_loss_decay is validated instead (ge=0, finite) in beb5028. For later readers: the earlier reply here describes a typed-list validator that the rebase removed.
| policy: | ||
| draft: | ||
| # One TTT pass per speculative token served by vLLM. | ||
| ttt_steps: 3 |
There was a problem hiding this comment.
1 action item.
This recipe adds 24 GPU-hours to the nightly budget, and the PR motivates multi-pass TTT by serving-time acceptance, but neither the description nor the nightly checks carry an acceptance or throughput number yet (the follow-up mentioned in #3699 (comment) has not landed).
Two asks:
- A small table in the PR description for
ttt_steps=3vsttt_steps=1from the same init: s/step, peak train memory,vllm/spec_acceptance_length,vllm/spec_acceptance_rate-pos-{1,2,3}, generation tok/s. These are already logged (), so no new instrumentation is needed. Also, the description's usage snippet showsRL/nemo_rl/models/generation/vllm/utils.py
Lines 730 to 734 in abfd277
loss_weight: 0.1and a draft optimizer atlr: 5e-5, while this recipe inheritsloss_weight: 1.0and no draft optimizer from its parent. Which one was measured? - The nightly asserts only
draft_lossshape (), and theRL/tests/test_suites/llm/grpo-qwen3-1.7b-1n8g-megatron-eagle3-ttt3.sh
Lines 49 to 51 in abfd277
0.8 * pass_3["1"]factor is not derived anywhere. A run where pass 3 learns nothing useful for serving can still pass. Since acceptance is logged, one line such asmean(data["vllm/spec_acceptance_length"], -5, -1) > <value from your run>would make this a regression test for what the feature is for.
Action: post ttt3 vs ttt1 numbers and add a vllm/spec_acceptance_length lower bound to the nightly checks.
There was a problem hiding this comment.
The description now opens with ttt3-vs-ttt1 tables: an 8B random-init pair from the initial revision (acceptance length 1.65 → 1.93) and both 1.7B nightlies at this head, including peak train memory (11.19 vs 11.40 GiB). The usage snippet now matches what was measured (loss_weight: 1.0, no draft optimizer). 4a6e762 adds mean(spec_acceptance_length, -5, -1) > 2.05 to the nightly (step 1 is 1.90, runs end at 2.26–2.29), and the 0.8 factor leaves margin over the observed 0.59–0.60 pass-3 loss ratio.
There was a problem hiding this comment.
C3 · No action needed — fixed in 42265a9. See the optional note on the 2.05 floor in the review body.
| # multi-pass config would otherwise surface as a KeyError far | ||
| # from the cause. | ||
| multi_pass = "student_logits_by_pass" in data_dict | ||
| if multi_pass != (ttt_steps > 1): |
There was a problem hiding this comment.
1 action item.
The two new guards have no tests: this config/forward consistency RuntimeError, and the split-path NotImplementedError (
RL/nemo_rl/models/policy/workers/megatron_policy_worker.py
Lines 1940 to 1947 in abfd277
tests/unit/models/policy/test_lm_policy_draft_guards.py::test_ttt_draft_with_sequence_packing_is_rejected (a stub cfg plus a data_dict with student_logits for this one; a stub worker with cfg["draft"].ttt_steps = 2 for the other).
Small question on the split-path guard: it fires inside the first train step, after model build and checkpoint load. Is there a reason it cannot sit next to the packing/SP checks in lm_policy.py, so a single-controller user fails at config time?
Action: add unit tests for the two guards; optionally hoist the split-path guard to config validation.
There was a problem hiding this comment.
6518b95 drops the config/forward check: the worker passes cfg["draft"].ttt_steps as draft_ttt_steps, so it compared the value with itself. The split-path guard now runs at the top of begin_train_step, tested in test_megatron_split_state.py. It can't sit in lm_policy.py because the policy config doesn't say which path runs: the single controller drives begin_train_step, while the data-plane GRPO trainer goes through train_presharded → batched train().
There was a problem hiding this comment.
C4 · No action needed — fixed in efa5683. The guard at the top of begin_train_step and its test cover this.
| logits = logits.transpose(0, 1).contiguous() | ||
| return logits | ||
|
|
||
| def _ttt_rotary_pos_emb(self, ttt_pass: int, seq_len: int) -> Optional[Tensor]: |
There was a problem hiding this comment.
1 action item (optional, low priority).
forward_ttt is only exercised through a mock (tests/unit/models/megatron/test_train.py::test_forward_with_ttt_draft_model_uses_forward_ttt), and nothing covers _ttt_rotary_pos_emb. The pass-d offset (rotary_pos_emb(seq_len + d - 1)[d - 1:]) must match vLLM's cache position at depth d; an off-by-one would still train to a low loss and only show up as acceptance decay at positions 2-3. A table-driven test with a fake rotary_pos_emb returning arange, asserting the slice for d in {1, 2, 3}, is cheap and needs no flash-attn. A second small test that pass-2 output carries grad back to pass-1 inputs (no detach) would lock in the contract stated in the forward_ttt docstring.
Action: (optional) unit tests for the rotary offsets and the no-detach gradient chain.
There was a problem hiding this comment.
Added the table-driven RoPE test in 4a6e762 (test_ttt_rotary_puts_pass_d_anchor_i_at_position_i_plus_d_minus_1, no GPU needed). I left out the gradient-chain test, since it would need flash-attn on a GPU and letting later passes backpropagate into earlier ones is the standard EAGLE-3 TTT setup, which SpecForge trains the same way.
There was a problem hiding this comment.
C5 · No action needed — RoPE offset test added in 42265a9. OK to skip the grad-chain test (GPU-only).
Extends the single-pass EAGLE-3 draft co-training path on Megatron to
multi-pass TTT (policy.draft.ttt_steps > 1):
- EagleModel.forward_ttt: sequential draft passes with the two-part
trunk+branch flash attention (exact joint-LSE merge) folded into
eagle.py, inference-aligned per-pass RoPE offsets, non-detached
hidden-state chain between passes.
- DraftTTTCrossEntropyLossFn: the multi-pass sibling of the single-pass
DraftCrossEntropyLossFn (which stays the ttt_steps == 1 loss,
including the packed layout). Per-pass slicing convention (pass d
predicts x_{i+d+1}); each pass is one DraftLossStats bin from the
streaming vocab-parallel soft-CE, weighted by
ttt_step_loss_decay**(d-1) and normalized by per-pass global valid
counts threaded from the worker (DP-only all-reduce). The d2t teacher
gather is tiled by policy.draft.token_chunk_size.
- The Megatron path now reads Eagle3DraftConfig's ttt_steps and
ttt_step_loss_decay, shared with the Automodel TTT path (default
ttt_steps=3; ttt_steps >= 1 is validated on the config). On Megatron,
ttt_steps > 1 rejects sequence packing (until NVIDIA-NeMo#4059), sequence
parallelism, and the split train-step path (its DraftStepState takes a
single loss bin). The single-pass eagle3 recipe pins ttt_steps: 1.
- Aux-layer ids resolved rank-independently
(resolve_draft_aux_layer_ids) and threaded into the hidden-state
capture, so PP stages cannot disagree about the P2P schedule.
- vLLM: un-share the drafter lm_head before loading a trained head
(refit would silently overwrite the target's head through vLLM's
share-on-identical-weights alias), preserving vLLM's weight-loader
attrs on the private copy.
- Nightly grpo-qwen3-1.7b-1n8g-megatron-eagle3-ttt3: the eagle3
nightly with ttt_steps=3, checking that the last pass learns (nightly
GPU-hour cap 5203 -> 5227).
Signed-off-by: Zili Wang <1348131731@qq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reject negative and non-finite decay factors: a negative factor flips the sign of every other pass's weight, and both enter the loss denominator directly. Signed-off-by: Zili Wang <1348131731@qq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…step Move the ttt_steps > 1 rejection from the first microbatch to the start of the split train step, with a test. Drop the forward/config consistency check in the loss setup: the only caller passes cfg ttt_steps as draft_ttt_steps, so it compared the value with itself. Signed-off-by: Zili Wang <1348131731@qq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…acceptance Assert pass d places anchor i at position i + d - 1, vLLM's position for the d-th speculative token. The ttt3 nightly now also requires the last-5 acceptance length above 2.05 (step 1: 1.90; runs end at 2.26-2.29). Signed-off-by: Zili Wang <1348131731@qq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a Multi-pass TTT section and the ttt_steps / ttt_step_loss_decay knobs, note that ttt_steps defaults to 3 so single-pass configs set it explicitly, and scope the sequence-packing support note to ttt_steps == 1. Signed-off-by: Zili Wang <1348131731@qq.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
seonjinn
left a comment
There was a problem hiding this comment.
Re-review of round 1. Thanks for the fixes. The split-path guard at begin_train_step, the RoPE offset test and the nightly gates address the round-1 asks.
- (C1) The draft
lm_headswap happens after CUDA graph capture, so the draft decode graphs may keep reading the target head.
Optional notes with no inline comment. Confidence is below the staging bar, so this is the author's call:
resolve_draft_aux_layer_idshas no unit test. Its three branches (checkpoint ids, config ids, modelopt default) can run on CPU with a patched AutoConfig.- The ttt3 nightly acceptance floor
> 2.05(line 52) is below the ttt1 acceptance quoted in the thread (2.24). Only the pass-3 loss ratio check separates ttt3 from ttt1. The floor would also not catch C1. - Pass chaining in
forward_tttis covered only by@requires_flash_attnGPU tests and the nightly. Which CI lane runs those? - The chunked TP gather and d2t path in loss/utils.py has no numeric test against the unchunked path.
- Megatron now honors the existing default
ttt_steps: 3. Megatron eagle3 users with sequence_parallel or packing get a startup ValueError until they setttt_steps: 1. This is worth one line in the release note. train/draft_lossfor ttt1 andtrain/draft_loss_pass_1for ttt>1 use different target offsets (documented inloss_functions.py), so curves across the two are not directly comparable.
Not executed locally (no GPU, Linux-only lockfile).
Merge note: this branch conflicts with latest main in nemo_rl/algorithms/loss/wrapper.py, tests/unit/models/policy/test_draft_config.py and tests/unit/test_recipes_and_test_suites.py.
Generated by Claude Code
| private_lm_head.parameters(), target_lm_head.parameters() | ||
| ): | ||
| private_param.__dict__.update(source_param.__dict__) | ||
| draft_model.lm_head = private_lm_head |
There was a problem hiding this comment.
C1 · 1 action item. TL;DR — draft decode CUDA graphs are captured at warm-up and keep the target head's address, so the new draft head that _unshare_draft_lm_head swaps in at refit is never read by them.
_unshare_draft_lm_head is called only from _load_draft_weights, which runs only on refit. That is after engine init and after CUDA graph capture. The swap rebinds draft_model.lm_head to a deep copy at a new address. It does not copy into the old storage. Graphs that were captured earlier still point at the target head.
The target head stays correct, so the PR's goal for the target model holds. The trained draft head goes to storage that the draft decode graphs never read. Acceptance stays above zero, so no check fails and the problem stays hidden. The default config has enforce_eager set to false, so graphs are on by default.
The docstring names SpecDecodeBaseProposer._maybe_share_lm_head, which is not the default path. The default V2 path aliases the head in load_eagle_model.
AI-1
Move the un-share to engine load, before CUDA graph capture, when draft co-training streams an lm_head. Capture then records the private head's address. Also change the docstring at line 1153 to name load_eagle_model (vLLM eagle/utils.py) as the default path. Please post the engine-log lines that show the runner and cudagraph mode from the ttt3 nightly. This is not a suggestion block because the fix moves code to engine init, outside this anchor.
vLLM v0.29.0 call chain behind the claim (not verified, no GPU environment)
- In the V2 runner speculator, draft decodes at positions above 0 use FULL_DECODE_ONLY graphs when the decode mode is FULL: speculator.py#L136-L138.
- Capture records the model forward, compute_logits and sample as one FULL graph: speculator.py#L157-L161.
- Eagle3
compute_logitsreadsself.lm_head: llama_eagle3.py#L339. - vLLM aliases the draft head to the target head when the draft has no
has_own_lm_headflag: eagle/utils.py#L12-L16 and eagle/utils.py#L83-L90. The flag is always missing underload_format=dummy, which is how NeMo-RL starts the engine before refit. - Draft prefill (position 0) graph mode was not confirmed.
- Not verified - no GPU environment.
| @model_validator(mode="after") | ||
| def _validate_ttt_pass_weights(self) -> Self: | ||
| if ( | ||
| self.ttt_pass_weights is not None | ||
| and len(self.ttt_pass_weights) != self.ttt_steps | ||
| ): | ||
| raise ValueError( | ||
| f"ttt_pass_weights must have ttt_steps={self.ttt_steps} entries, " | ||
| f"got {len(self.ttt_pass_weights)}." | ||
| ) | ||
| return self |
There was a problem hiding this comment.
C2 · No action needed — resolved by removal. ttt_pass_weights is gone at this head, and ttt_step_loss_decay is validated instead (ge=0, finite) in beb5028. For later readers: the earlier reply here describes a typed-list validator that the rebase removed.
| policy: | ||
| draft: | ||
| # One TTT pass per speculative token served by vLLM. | ||
| ttt_steps: 3 |
There was a problem hiding this comment.
C3 · No action needed — fixed in 42265a9. See the optional note on the 2.05 floor in the review body.
| # multi-pass config would otherwise surface as a KeyError far | ||
| # from the cause. | ||
| multi_pass = "student_logits_by_pass" in data_dict | ||
| if multi_pass != (ttt_steps > 1): |
There was a problem hiding this comment.
C4 · No action needed — fixed in efa5683. The guard at the top of begin_train_step and its test cover this.
| logits = logits.transpose(0, 1).contiguous() | ||
| return logits | ||
|
|
||
| def _ttt_rotary_pos_emb(self, ttt_pass: int, seq_len: int) -> Optional[Tensor]: |
There was a problem hiding this comment.
C5 · No action needed — RoPE offset test added in 42265a9. OK to skip the grad-chain test (GPU-only).
What does this PR do ?
Extends the single-pass EAGLE-3 draft co-training path on the Megatron policy backend to multi-pass TTT training (
policy.draft.ttt_steps > 1), co-trained with the policy during GRPO and refit into vLLM alongside the policy weights. The Automodel backend's TTT path landed in #3960; both backends now read the samettt_steps/ttt_step_loss_decayconfig.Issues
Part of #3698 (1/5). Stacked series: this PR → #3712 (DFlash) → #3713 (DSpark) → #4059 (sequence packing + CP) → #4060 (PP).
Usage
Before your PR is "Ready for review"
Pre checks:
Additional Information
Rebased onto main after the typed speculator contract (#3701), streaming soft-CE (#3703), split-step normalization (#3706), draft optimizer overrides (#3707) and Automodel draft co-training (#3960) landed; the pieces those cover were dropped from this PR. Main pieces:
docs/guides/eagle3-speculative-decoding.md.EagleModel.forward_ttt): sequential draft passes with a two-part trunk+branch flash attention (exact joint-LSE merge), inference-aligned per-pass RoPE offsets, non-detached hidden-state chain between passes.DraftTTTCrossEntropyLossFn— the multi-pass sibling ofDraftCrossEntropyLossFn, which stays thettt_steps == 1loss (including the packed layout) — slices each pass (pass d predicts x_{i+d+1}) and feeds it to the streaming soft-CE as oneDraftLossStatsbin, weighted byttt_step_loss_decay ** (d - 1)and normalized by per-pass global valid counts (DP-only all-reduce).Eagle3DraftConfig.ttt_steps/ttt_step_loss_decay, which feat(automodel): speculative decoding draft co-training #3960 added for the Automodel path (defaultttt_steps: 3; this PR validatesttt_steps >= 1and a finite, non-negative decay). On Megatron,ttt_steps > 1rejects sequence packing (until feat(spec-decode): context parallelism and sequence packing for draft co-training (Megatron) #4059) and sequence parallelism, andbegin_train_steprejects it on the split train-step path (itsDraftStepStatetakes a single loss bin). Megatron configs that leavettt_stepsunset now train 3 passes, so the single-pass eagle3 recipe pinsttt_steps: 1.ttt_steps == 1routes to the existing single-pass forward and loss; it is not a one-pass run of the TTT loss.grpo-qwen3-1.7b-1n8g-megatron-eagle3-ttt3, the eagle3 nightly withttt_steps: 3. Beyond the single-pass checks it requires the pass-3 loss to fall below 0.8x its step-1 value (observed 0.59-0.60x) and the last-5 acceptance length to stay above 2.05 (step 1: 1.90; observed 2.26 and 2.29 in two runs). Nightly GPU-hour cap 5203 → 5227.Smoke-tested on a Qwen3-8B GRPO math recipe. Both eagle3 nightlies (single-pass and ttt3) pass end to end on a vLLM 0.29 container (50 steps, 1n8g), re-run on the current head.
A suspected TP grad all-reduce gap on the (unexercised) split-API train-step path was found during this work; it is tracked separately with its regression test in #3716 rather than bundled here.
Evaluation verification
Perf:

ttt_steps=3vsttt_steps=1Same init and recipe in each pair; only
ttt_stepsdiffers.Qwen3-8B GRPO math, randomly initialized EAGLE-3 draft (TP2,
loss_weight: 1.0, draft lr 1e-4, 3 speculative tokens; mean over steps 169-178):vllm/spec_acceptance_lengthvllm/spec_acceptance_rate-pos-{1,2,3}Single-pass training never sees the draft's own outputs, so positions 2-3 barely train; ttt3 trains every served position. Measured on this PR's initial revision: the loss has since moved onto main's streaming soft-CE (#3703), while the TTT forward and attention are unchanged; the ttt3 run's first 25 steps predate the inference-aligned RoPE offset. Peak memory was not logged in these runs.
Qwen3-1.7B, the two eagle3 nightlies at this head (
AngelSlim/Qwen3-1.7B_eagle3init,loss_weight: 1.0, no draft optimizer, 1n8g; mean over the last 5 of 50 steps):vllm/spec_acceptance_lengthvllm/spec_acceptance_rate-pos-{1,2,3}max_memory_allocated, GiB)From a pretrained drafter the gap is small at 50 steps; this pair is the cost check. Peak memory is the max over ranks of
torch.cuda.max_memory_allocated()acrosstrain(), from a temporary probe that is not part of this PR.🤖 Generated with Claude Code