Skip to content

Bump Megatron-LM to miles-main-20260819 (latest NVIDIA dev) - #2673

Merged
yueming-yuan merged 32 commits into
mainfrom
megatron-bump-20260819
Aug 24, 2026
Merged

yueming-yuan merged 32 commits into
mainfrom
megatron-bump-20260819

Conversation

@yueming-yuan

Copy link
Copy Markdown
Collaborator

ci-megatron-pr: miles-main-20260819

What

Bump the Megatron-LM that miles depends on from the stale 2026-02-14 fork point to latest
NVIDIA/Megatron-LM dev (3f49de20d, 2026-08-19). Supersedes #1466 (June bump, never merged).

Scope: base bump only. DeepSeek-V4 and true-on-policy are explicitly disabled and land in
dedicated follow-up PRs (DSv4 first, then true-on-policy).

Megatron side — radixark/Megatron-LM:miles-main-20260819

nvidia/dev HEAD + 20 commits: 14 rebased from the June bump branch (miles-main-20260622) +
6 newly ported miles-main commits (#21 witness, #63 NVMe streaming, #68 fp32 MoE, #74 ROCm
dist-ckpt, #72 base64 te-precision-config, #82 LayerWise expert ownership).

Dropped as upstreamed into dev since June (verified feature-by-feature):

Miles side (this PR)

  • Compat fixes re-derived on main: gloo arg rename (4 sites), drop moe_use_legacy_grouped_gemm,
    MTP transformer_layer→mtp_model_layer in all converters, MTP loss tracker
    reduce_loss_in_tracker, name= kwarg in custom attention, mtp_detach_heads adaptation,
    dumper anchors → _forward_mlp_output_with_bias, LoRA buffer patch → ddp_config flags.
  • Explicit disables: megatron backend + --true-on-policy-mode raises NotImplementedError;
    test_model_provider_true_on_policy.py skipped; DSv4 4-layer e2e disabled via
    register_cuda_ci(disabled=...). fsdp true-on-policy is unaffected.
  • docker/Dockerfile: MEGATRON_BRANCH=miles-main-20260819.

Follow-ups (not this PR)

  1. DSv4 PR: the dsv4 attention variant + --dsv4-impl {miles,megatron} dual backend + R3
    threshold sign-off.
  2. True-on-policy PR: the SGLang-matching megatron backend + miles_megatron_plugins packaging.

Megatron dev reimplemented PR #6's MTP-in-RL support natively:
- process_mtp_loss derives MTP labels from input_ids when labels is None (RL).
- config.mtp_detach_heads detaches output head + MTP embedding gradients.
So on the miles side: set config.mtp_detach_heads=True when enable_mtp_training,
and stop passing the now-unsupported mtp_kwargs to GPTModel.forward (labels=None
+ input_ids derivation is equivalent to mtp_labels=batch['tokens']).
… lora test mock

Matches the dev Megatron arg rename used in model.py (use_gloo_process_groups).
…put_with_bias

New dev split the MLP into _forward_mlp_output_with_bias + _forward_post_mlp and dropped
_sglang_pre_mlp_residual (true-on-policy was deferred). Anchor pre_mlp_residual on the
unconditional nvtx_range_push("mlp") (residual set in all branches); keep pre_mlp_layernorm_output
and mlp_output_with_bias anchors. Will re-verify after the true-on-policy migration.
…bias

The megatron true-on-policy merge relocated the _sglang_pre_mlp_residual extraction
out of _forward_mlp into _forward_mlp_output_with_bias, so the BSHD source-patcher
anchor 'residual = getattr(self, "_sglang_pre_mlp_residual", hidden_states)' no
longer exists. Mirror the THD config: target _forward_mlp_output_with_bias and
prepend the pre_mlp_residual dump at the mlp nvtx push (matches the new structure).
Fixes test_run_megatron + test_miles_dumper source-patch failures.
…_module)

New dev's build_module passes name= to every spec-built module (megatron base
Attention.__init__ accepts it). miles' custom attention classes overriding __init__
must accept + thread it: DSAMultiLatentAttention/DSAMLASelfAttention (glm5, DeepSeek-V3.2),
HuggingfaceAttention base, qwen3_5/qwen3_next Attention. Mirrors the DeepSeekV4Attention
name fix. Fixes 'DSAMLASelfAttention.__init__() got an unexpected keyword argument name'.
…lders

New dev removed the moe_use_legacy_grouped_gemm argument and the legacy grouped-GEMM
concept (grouped_mlp_modules now takes only moe_use_grouped_gemm). miles' model_provider
and glm4 spec builders read args.moe_use_legacy_grouped_gemm, which no longer exists ->
AttributeError on the Namespace. Drop the kwarg (dev spec fns take **kwargs / the new
grouped path is the only one).
New Megatron renamed MultiTokenPredictionLayerSubmodules.transformer_layer ->
mtp_model_layer, so MTP weight names are now mtp.layers.{i}.mtp_model_layer.*.
miles' MTP weight converters (mcore<->hf) hardcoded 'transformer_layer' ->
'Unknown MTP component' on rollout weight export (test_mimo_7B_mtp_only_grad).
Accept both names across export (mimo/qwen3_next/deepseekv3/glm4moe/qwen3_5 + fp8/
mxfp8/nvfp4 quantizers) and mbridge import (mimo/qwen3_next/qwen3_5/glm4moe_lite/glm4moe).
New dev rewrote MTPLossLoggingHelper: it stores loss_sums/num_tokens (or loss_values)
and only computes tracker['values'] after reduce_loss_in_tracker() (which also does the
cross-rank all-reduce). miles read tracker['values'] directly (never populated) ->
'values' not in tracker -> mtp_losses unbound -> UnboundLocalError at extra_metrics.
Call reduce_loss_in_tracker(), read values/loss_values, init mtp_losses=None + guard
the metric. Fixes test_mimo_7B_mtp_only_grad past weight-export.
New Megatron reads disable_{param,grad}_buffers_cpu_backup from
DistributedDataParallelConfig; _ParamAndGradBuffer no longer takes them
as ctor kwargs.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@yueming-yuan
yueming-yuan force-pushed the megatron-bump-20260819 branch from 2284788 to 0280147 Compare August 20, 2026 03:42
@yueming-yuan yueming-yuan added the bypass-fastfail Disable CI fast-fail: run all stages and all tests to surface every failure label Aug 20, 2026
Rebuilds were driven by whether the PR diff touched docker paths, so every
rerun of a Dockerfile-touching PR paid for an identical multi-arch build.

Hash the build inputs, stamp the hash on the published tag as a label, and
compare against it: a PR keeps one pr-<num> tag and rebuilds it only when the
inputs actually move, when the tag is missing, or on the one-shot
rebuild-ci-image label.

resolve-ci-image now selects the PR image whenever that tag is current rather
than only when this run built it, so reusing an image does not silently fall
back to dev.
Also scope pull-requests: write to the docker-build call instead of the whole
workflow; only that job removes the one-shot label.
TopKRouter.forward gained packed_seq_params upstream, which MoELayer passes
positionally. Separately, the plain bridge provider never forwarded
gradient_accumulation_fusion, so --no-gradient-accumulation-fusion could not
reach models built through it (the LoRA bridge path already synced it).
# Conflicts:
#	.github/workflows/_build-pr-ci-image.yml
Megatron now resolves te_general_gemm, which it could not on TE 2.17 before the
bump, and TE multiplies in the operand dtype -- so --moe-router-dtype fp32 became
a bf16 multiply. The last ulp is enough for top-k to pick a different expert, and
on qwen3.5 that flipped one token in layer 0 and cascaded to a 15% logprob drift.
25 model scripts ask for fp32 routing, so default it here rather than per model.
@guapisolo

Copy link
Copy Markdown
Collaborator

@codex review

@guapisolo

Copy link
Copy Markdown
Collaborator

@claude review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6ddbd76bb4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docker/Dockerfile

ARG MEGATRON_REPO=radixark/Megatron-LM
ARG MEGATRON_BRANCH=miles-main
ARG MEGATRON_BRANCH=miles-main-20260819

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update every Megatron consumer to the new branch

Changing only the CUDA Dockerfile leaves several automated contexts on the incompatible miles-main revision: _run-ci.yml:140 and _run-cpu-ci.yml:80 default to it when no PR-body override exists, release-branch-cut.yml:101 locks releases from it, and Dockerfile.rocm:24 still builds it directly. This commit now imports the new tokenizer module and reads renamed fields such as args.use_gloo_process_groups, so nightly/post-merge CI, release images, and ROCm images will pair the new Miles code with the pre-bump Megatron API and fail during import or initialization. Update those dependency defaults and the release lock source, or retain compatibility with the old API.

Useful? React with 👍 / 👎.

@Zhichenzzz

Copy link
Copy Markdown
Collaborator

@yueming-yuan did you test the dist_muon path for the new branch? worthy having it

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this PR and didn't find any bugs. Because it's a large Megatron-LM version bump with numerics-sensitive compat adaptations spanning the training loop, LoRA, MTP, and weight quantization/conversion, a human look would still be worthwhile.

What was reviewed: the forced moe_router_use_torch_mm=True/trust_remote_code=True and new true_on_policy_mode guard in arguments.py; the enable_gloo_process_groups → use_gloo_process_groups rename across model.py/initialize.py/multi_lora_optimizer.py; the ddp_config-based LoRA buffer patch in lora_utils.py; the MTP loss-tracker (reduce_loss_in_tracker, values/loss_values) and mtp_detach_heads changes in model.py/model_provider.py; the transformer_layer→mtp_model_layer renaming applied consistently across update_weight/common.py and the fp8/mxfp8/nvfp4 quantizers; and the inkling plugin's move to partial-based ModuleSpec construction and mark_keep_in_fp32.

Extended reasoning...

Overview

This PR bumps the vendored Megatron-LM pin from a February 2026 fork point to the August 2026 NVIDIA dev HEAD, and adapts miles' megatron_utils backend and several model plugins (glm4, glm5, qwen3_5, qwen3_next, mimo, inkling) to the new upstream API surface. Substantial additional compat work landed across ~30 commits after this bug-hunting system's earlier review of this PR (which occurred very early, right after the initial squash of ported commits): moved vocab_size_with_padding import location, forced trust_remote_code=True and moe_router_use_torch_mm=True (to counter a new TE bf16 router-gemm default that increases logprob diff vs. old fp32 gemm), an explicit NotImplementedError for true_on_policy_mode on the megatron backend, a config-flag rename (enable_gloo_process_groups→use_gloo_process_groups), a ddp_config-based rewrite of the _ParamAndGradBuffer LoRA monkeypatch, MTP loss-tracker API changes (reduce_loss_in_tracker, values/loss_values key), mtp_detach_heads wiring guarded with getattr for contexts lacking the miles-specific arg, MTP submodule renaming (transformer_layer→mtp_model_layer) applied consistently in weight-sync regex and quantizer name-stripping, and an inkling plugin rewrite to the new partial-based ModuleSpec construction with a mark_keep_in_fp32 helper.

Security risks

None identified. trust_remote_code=True is now forced when building the tokenizer, which executes remote code from the tokenizer repo, but this is an intentional, disclosed compatibility change for the new Megatron tokenizer loading path and consistent with existing patterns in ML training frameworks; there is no user-controlled input reaching this path beyond the already-trusted model/checkpoint config.

Level of scrutiny

This warrants above-baseline scrutiny: it is a large dependency bump (39 files) touching numerics-sensitive code paths (router gemm precision, MTP loss aggregation, distributed optimizer buffer flags, LoRA parameter buffer patches) where subtle behavioral regressions would not necessarily surface as hard failures but as silent accuracy/training-quality drift. The PR description itself documents several compensating fixes for exactly this kind of regression (the router-gemm precision fix), which suggests the space is fragile and worth a second set of human eyes, especially since correctness here is best validated via the e2e CI numeric-diff tests referenced in the PR (some of which — Nemotron-3 Ultra, GLM5.1/5.2 LoRA, DeepSeek-V4 4-layer — are explicitly disabled pending follow-up work) rather than through static code reading alone.

Other factors

The renames and pattern updates (gloo flag, MTP layer name, quantizer stripping) were applied consistently across all call sites I checked, and the changes for run_megatron debug worker contexts correctly guard newly-introduced arg reads with getattr(..., False) defaults to avoid AttributeError in contexts lacking miles-specific args. This run's automated bug hunt reported no findings and no candidate issues were specifically ruled out, so this defer note reflects the change's inherent scope/risk rather than any unresolved concern.

@guapisolo

Copy link
Copy Markdown
Collaborator

[P2] Avoid re-scaling per-token MTP loss by the microbatch count

At miles/backends/megatron_utils/model.py:804-809, MTPLossLoggingHelper.reduce_loss_in_tracker() already computes sum(loss) / sum(tokens) when calculate_per_token_loss is enabled. The pinned Megatron training path correspondingly uses mtp_loss_scale = 1.0 for that mode, but Miles still unconditionally multiplies the result by 1 / num_microbatches[step_id].

With --enable-mtp-training --calculate-per-token-loss and N microbatches, the reported mtp_loss and the value passed to check_mtp_loss are therefore N times too small. Please apply the microbatch averaging only in the non-per-token mode.

MTPLossLoggingHelper already normalizes per-token loss across all tracked
tokens. Use unit scaling in that mode while preserving legacy microbatch
averaging.
@guapisolo

Copy link
Copy Markdown
Collaborator

Prospective tip 59cc3c11f3ca851db37c81fc316f6facbeda1318 will stop applying microbatch averaging after MTPLossLoggingHelper has already produced a per-token-normalized MTP loss, while preserving legacy averaging. The candidate passed the target file's full pre-commit hooks on miles-h200-dev-2 and the synced AST check (per_token=1.0, legacy=0.25 for four microbatches).

@guapisolo

Copy link
Copy Markdown
Collaborator

novita 2 has been fixed. re-kick off ci

@guapisolo

Copy link
Copy Markdown
Collaborator

Codex review

Reviewed against Miles PR head c292bee42b219fa4921e33a540cdbd55fa73aa18 and radixark/Megatron-LM:miles-main-20260819 at 235952df607b3820716e5e67728a5ab470ca33ae.

Review 2 — [P1] Keep the shared checkpoint merge helper independent of training globals

4c725b138 added an unconditional get_args().low_memory_resume read to megatron/core/transformer/utils.py::cat_with_oom_fallback. This function is a shared MCore ShardedTensorFactory.merge_fn, used by SSM, QKV attention, and GLU MLP checkpoint factories and reached from the public distributed-checkpoint load path. A standalone MCore, Megatron-Bridge, or checkpoint-conversion caller is not required to initialize megatron.training globals, so it now fails before attempting torch.cat with:

AssertionError: args is not initialized.

This is introduced relative to the new upstream baseline: the parent helper had no training-global dependency. Old miles-main had a similar read only in an MLP-private callback; 4c725b138 expands it into the generic helper. Targeted devbox runs reproduce the regression in both existing cases:

  • test_ssm_split_tensor_factory_oom_is_handled[gated_delta_net]
  • test_ssm_split_tensor_factory_oom_is_handled[mamba_mixer]

Please make an uninitialized args state equivalent to low_memory_resume=False, preserving the normal merge and existing OOM fallback. Longer term, the force-CPU decision should be passed through a core-scoped checkpoint context instead of introducing a megatron.core -> megatron.training dependency.

Review 3 — [P1 if reverse-picked; non-blocking for PR #2673] Do not apply 95e3a6d to old miles-main

95e3a6d is correct on the dated branch because upstream ceaa6c5d7e rebuilds model_param_group_index_map after the optimizer's fp32-first parameter order has been installed. Old miles-main does not contain ceaa6c5d7e; its only correct rebuild is the implementation introduced by cf014f903 inside _build_model_and_main_param_groups.

Reverse-picking 95e3a6d onto old produces content conflicts. Resolving those conflicts by accepting the dated deletion removes old's only rebuild. Mixed fp32 and bf16/fp16 parameters in one optimizer group would then use grad-buffer order to index an fp32-first optimizer group, causing checkpoint save/load to read another parameter's main parameter or Adam state. Different shapes normally crash; equal shapes can silently associate state with the wrong parameter. The later 479bad6 restores only the fp32 leaf detach and does not restore this map.

No change is required for PR #2673 for this item: its dated target retains the upstream post-allocation rebuild. This is a backport guard—skip 95e3a6d when moving dated follow-ups onto old miles-main, unless ceaa6c5d7e is ported first in the same atomic change.

@yueming-yuan
yueming-yuan merged commit 580040b into main Aug 24, 2026
27 of 49 checks passed
@yueming-yuan
yueming-yuan deleted the megatron-bump-20260819 branch August 24, 2026 19:37
yueming-yuan added a commit that referenced this pull request Aug 24, 2026
#2673 was squash-merged, so main's base bump and this branch's copy of it are
unrelated histories. Three resolutions git could not get right on its own:

- docker/Dockerfile: main is net-unchanged from the merge base (bump set
  miles-main-20260819, #2734 set it back), so git kept this branch's
  miles-main-20260819 — a branch that no longer exists on the Megatron remote.
  Take miles-main.
- test_deepseek_v4_flash_4layer_ci.py: the inverse — this branch is
  net-unchanged, so git kept main's disabled=. Re-enable it; that is this PR.
- model.py mtp_loss_scale: this branch predates 59cc3c1, so its side is the
  unfixed line. Take main's.
yueming-yuan added a commit that referenced this pull request Aug 24, 2026
Picks up the Megatron bump (#2673) and its follow-up (#2734), which points
MEGATRON_BRANCH back at miles-main after miles-main-20260819 disappeared from
the Megatron remote.
shawnghu added a commit to shawnghu/miles that referenced this pull request Oct 2, 2026
…aining is off

Since the Megatron bump in radixark#2673, Megatron derives MTP labels from input_ids whenever
labels=None and backprops the MTP loss on every training forward once the MTP block
exists. The bridge builds that block from the HF config (mtp_num_hidden_layers /
num_nextn_predict_layers), so every bridge-mode run on an MTP checkpoint trained the
decoder (and LoRA adapters) on an extra next-next-token loss over its own rollouts,
independent of the RL loss, even without --enable-mtp-training.

- full-model bridge: build the MTP block only with --enable-mtp-training
- LoRA bridge: never build it; reject --enable-mtp-training (the MTP head is frozen
  base weights and is not synced, so it could not train, while its loss still reached
  the adapters undetached). Previously only multi-LoRA did this (radixark#3584).
- raw path: keep the block for checkpoint compatibility but zero its loss scale when
  --mtp-num-layers is set without --enable-mtp-training
- test_bridge_mtp_detachment: assert the block is not built when MTP training is off

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fastfail Disable CI fast-fail: run all stages and all tests to surface every failure run-ci-image

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants