Repository navigation
DeepSeek-V4 support-v2: native megatron implementation as option - #2706
Merged
Merged
Conversation
Megatron dev reimplemented PR #6's MTP-in-RL support natively: - process_mtp_loss derives MTP labels from input_ids when labels is None (RL). - config.mtp_detach_heads detaches output head + MTP embedding gradients. So on the miles side: set config.mtp_detach_heads=True when enable_mtp_training, and stop passing the now-unsupported mtp_kwargs to GPTModel.forward (labels=None + input_ids derivation is equivalent to mtp_labels=batch['tokens']).
… lora test mock Matches the dev Megatron arg rename used in model.py (use_gloo_process_groups).
…put_with_bias
New dev split the MLP into _forward_mlp_output_with_bias + _forward_post_mlp and dropped
_sglang_pre_mlp_residual (true-on-policy was deferred). Anchor pre_mlp_residual on the
unconditional nvtx_range_push("mlp") (residual set in all branches); keep pre_mlp_layernorm_output
and mlp_output_with_bias anchors. Will re-verify after the true-on-policy migration.
…bias The megatron true-on-policy merge relocated the _sglang_pre_mlp_residual extraction out of _forward_mlp into _forward_mlp_output_with_bias, so the BSHD source-patcher anchor 'residual = getattr(self, "_sglang_pre_mlp_residual", hidden_states)' no longer exists. Mirror the THD config: target _forward_mlp_output_with_bias and prepend the pre_mlp_residual dump at the mlp nvtx push (matches the new structure). Fixes test_run_megatron + test_miles_dumper source-patch failures.
…_module) New dev's build_module passes name= to every spec-built module (megatron base Attention.__init__ accepts it). miles' custom attention classes overriding __init__ must accept + thread it: DSAMultiLatentAttention/DSAMLASelfAttention (glm5, DeepSeek-V3.2), HuggingfaceAttention base, qwen3_5/qwen3_next Attention. Mirrors the DeepSeekV4Attention name fix. Fixes 'DSAMLASelfAttention.__init__() got an unexpected keyword argument name'.
…lders New dev removed the moe_use_legacy_grouped_gemm argument and the legacy grouped-GEMM concept (grouped_mlp_modules now takes only moe_use_grouped_gemm). miles' model_provider and glm4 spec builders read args.moe_use_legacy_grouped_gemm, which no longer exists -> AttributeError on the Namespace. Drop the kwarg (dev spec fns take **kwargs / the new grouped path is the only one).
New Megatron renamed MultiTokenPredictionLayerSubmodules.transformer_layer ->
mtp_model_layer, so MTP weight names are now mtp.layers.{i}.mtp_model_layer.*.
miles' MTP weight converters (mcore<->hf) hardcoded 'transformer_layer' ->
'Unknown MTP component' on rollout weight export (test_mimo_7B_mtp_only_grad).
Accept both names across export (mimo/qwen3_next/deepseekv3/glm4moe/qwen3_5 + fp8/
mxfp8/nvfp4 quantizers) and mbridge import (mimo/qwen3_next/qwen3_5/glm4moe_lite/glm4moe).
New dev rewrote MTPLossLoggingHelper: it stores loss_sums/num_tokens (or loss_values) and only computes tracker['values'] after reduce_loss_in_tracker() (which also does the cross-rank all-reduce). miles read tracker['values'] directly (never populated) -> 'values' not in tracker -> mtp_losses unbound -> UnboundLocalError at extra_metrics. Call reduce_loss_in_tracker(), read values/loss_values, init mtp_losses=None + guard the metric. Fixes test_mimo_7B_mtp_only_grad past weight-export.
New Megatron reads disable_{param,grad}_buffers_cpu_backup from
DistributedDataParallelConfig; _ParamAndGradBuffer no longer takes them
as ctor kwargs.
…rd-only run-megatron skips DDP wrap
Rebuilds were driven by whether the PR diff touched docker paths, so every rerun of a Dockerfile-touching PR paid for an identical multi-arch build. Hash the build inputs, stamp the hash on the published tag as a label, and compare against it: a PR keeps one pr-<num> tag and rebuilds it only when the inputs actually move, when the tag is missing, or on the one-shot rebuild-ci-image label. resolve-ci-image now selects the PR image whenever that tag is current rather than only when this run built it, so reusing an image does not silently fall back to dev.
Also scope pull-requests: write to the docker-build call instead of the whole workflow; only that job removes the one-shot label.
…' into megatron-bump-20260819
New dev returns functools.partial builders for the mlp and shared-experts submodules instead of ModuleSpec, and MoELayer now passes name= to the shared-experts builder.
Megatron now rejects the combination: fake QAT swaps in straight-through weight tensors while TE's fused wgrad accumulation writes main_grad onto the original ones, so the quantized weights' gradients would be dropped.
Megatron's dsa variant now builds AbsorbedMLASelfAttention, which reads linear_kv_up_proj.weight directly; the pinned Bridge's LoRALinear has no such property. Upstream Bridge added one that returns the LoRA-effective weight.
Megatron's Mamba mixer exposes conv1d as conv1d_weight/conv1d_bias parameters; the pinned Bridge maps only the conv1d.* module names, so the weight sync skips them silently. Upstream Bridge accepts both spellings.
New Megatron renamed the MTP submodule to mtp_model_layer, so MTP expert weights missed the expert-offset rewrite and ranks broadcast differing tensor sets during weight sync, deadlocking the EP collectives.
Megatron now gates the HF tokenizer on args.trust_remote_code, which miles never set, so checkpoints shipping custom tokenizer code (Kimi) failed to load. miles already passes trust_remote_code=True everywhere it loads a checkpoint itself.
The private _keep_fp32 attribute was only honoured by the fp32 snapshot that rode in on the DSv4 megatron commit, which the base bump defers. Upstream now owns this: mark_keep_in_fp32 exempts the tensor from Float16Module's cast, so the scale survives the bf16 round trip through the weight sync.
TopKRouter.forward gained packed_seq_params upstream, which MoELayer passes positionally. Separately, the plain bridge provider never forwarded gradient_accumulation_fusion, so --no-gradient-accumulation-fusion could not reach models built through it (the LoRA bridge path already synced it).
# Conflicts: # .github/workflows/_build-pr-ci-image.yml
Megatron now resolves te_general_gemm, which it could not on TE 2.17 before the bump, and TE multiplies in the operand dtype -- so --moe-router-dtype fp32 became a bf16 multiply. The last ulp is enough for top-k to pick a different expert, and on qwen3.5 that flipped one token in layer 0 and cascaded to a 15% logprob drift. 25 model scripts ask for fp32 routing, so default it here rather than per model.
Same recipe as the miles-impl test with the megatron implementation selected; the PyTorch DSA fallback is used until the CI image ships flash_mla and cudnn-frontend, so the fused-kernel path stays uncovered.
yueming-yuan
force-pushed
the
dsv4-dual-backend
branch
from
August 27, 2026 07:17
2bb54d6 to
4b651a4
Compare
The --dsv4-* aliases duplicated flags Megatron already exposes (--csa-*, --o-groups, --o-lora-rank, --num-residual-streams, --mhc-sinkhorn-iterations, --moe-n-hash-layers); the plugin now only declares --dsv4-impl. Also regenerates the model_args snapshots, stale since 43c5f1c.
…kend-drop-colocate
yueming-yuan
force-pushed
the
dsv4-dual-backend
branch
from
August 27, 2026 21:29
ea4c4bb to
03856d9
Compare
Zhichenzzz
approved these changes
Aug 31, 2026
Zhichenzzz
left a comment
Collaborator
There was a problem hiding this comment.
Most looks good! just small comments
| # RL rollout-routing-replay (R3) seam for the sparse-attention indexer topk: lets the miles | ||
| # indexer_replay_manager record (on rollout) / replay (on train forward) the top-k KV picks, | ||
| # mirroring the MoE routing-replay seam. No-op unless the manager is enabled. | ||
| indexer_replay_manager.register_to_module(self, "indexer_replay", stream_idx=layer_id) |
Collaborator
There was a problem hiding this comment.
(maybe later we should opt this indexer replay, since now it is easy to cause host oom / and long-latency. and this will be very help for the following models, e.g., glm, qwen, ...
| @@ -1 +1 @@ | |||
| import torch | |||
Collaborator
There was a problem hiding this comment.
rename the file into deepseek_v4?
|
|
||
|
|
||
| def convert_deepseekv4_to_hf(args, name, param): | ||
| def _packed_alphas(name: str, param, bucket): |
Collaborator
There was a problem hiding this comment.
claude code says this will harm async train, but i dont agree (maybe quick check
Collaborator
Author
There was a problem hiding this comment.
just checked, this should not harm the training
| # Non-interleaved indexers keep wk as a standalone FP8 parameter in SGLang. | ||
| fp8_param_names.extend( | ||
| [ | ||
| "self_attention.wk.weight", |
| @@ -1,2 +1,2 @@ | |||
| """ | |||
| DeepSeek V4 training script. | |||
Collaborator
There was a problem hiding this comment.
do we also need to modify amd script into the latest impl?
xiaohong42
added a commit
to xiaohong42/miles
that referenced
this pull request
Sep 14, 2026
Rebased onto current main: the original diff was written before radixark#2706 renamed the attention submodules, so it read against self.q_norm / self.wq_b / self.wkv. It still applied, because git matched the three unchanged q_fp32 lines rather than the renamed context, but the diff no longer described the file a reviewer would open. The change itself is unmodified. Not bit-identical on GPU, which the previous description did not say. Against the manual fp32-intermediate version, bf16 differs by one ULP on about 4.1e-6 of elements (measured over 4096x64x512 on gfx942). Both agree with an fp64 reference to the same maximum relative error, 3.891e-3, which is bf16's own quantisation limit -- so this is rounding-tie direction, not a precision change. Callers that compare logprobs bit-for-bit across a version boundary should know.
xiaohong42
added a commit
to xiaohong42/miles
that referenced
this pull request
Sep 20, 2026
Rebased onto current main: the original diff was written before radixark#2706 renamed the attention submodules, so it read against self.q_norm / self.wq_b / self.wkv. It still applied, because git matched the three unchanged q_fp32 lines rather than the renamed context, but the diff no longer described the file a reviewer would open. The change itself is unmodified. Not bit-identical on GPU, which the previous description did not say. Against the manual fp32-intermediate version, bf16 differs by one ULP on about 4.1e-6 of elements (measured over 4096x64x512 on gfx942). Both agree with an fp64 reference to the same maximum relative error, 3.891e-3, which is bf16's own quantisation limit -- so this is rounding-tie direction, not a precision change. Callers that compare logprobs bit-for-bit across a version boundary should know.
xiaohong42
added a commit
to xiaohong42/miles
that referenced
this pull request
Sep 21, 2026
Rebased onto current main: the original diff was written before radixark#2706 renamed the attention submodules, so it read against self.q_norm / self.wq_b / self.wkv. It still applied, because git matched the three unchanged q_fp32 lines rather than the renamed context, but the diff no longer described the file a reviewer would open. The change itself is unmodified. Not bit-identical on GPU, which the previous description did not say. Against the manual fp32-intermediate version, bf16 differs by one ULP on about 4.1e-6 of elements (measured over 4096x64x512 on gfx942). Both agree with an fp64 reference to the same maximum relative error, 3.891e-3, which is bf16's own quantisation limit -- so this is rounding-tie direction, not a precision change. Callers that compare logprobs bit-for-bit across a version boundary should know. (cherry picked from commit f9f975a)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ci-megatron-pr: dsv4-dual-backend
What
Bring DeepSeek-V4 back on the bumped Megatron, and make the training implementation an explicit
choice. Second of the three bump PRs; stacked on #2673 (base bump), which disabled DSv4.
--dsv4-impl {miles,megatron}selects a whole path, at config+spec build time:miles(default)megatrondsv4(ported PR #28 slice)dsv4_hybrid(dev-native)enable_hyper_connectionsUnsupported combinations raise, they do not fall back — June's
MILES_DSV4_ATTENTION_BACKENDsilently forced miles on TP>1 and that behaviour is retired.
Megatron side —
radixark/Megatron-LM:dsv4-dual-backendmiles-main-20260819+ 6 commits: the miles DSv4 attention variant, hash-routing under TP + MoEfreeze knobs, and four fixes found by running it:
tid2eidwas a buffer, not a parameter — miles' weight sync materializes full tensors from theparameter buffer, so it pushed an all-zero routing table into sglang and the engine became a
different model. The weight checker caught this on the first run.
activation_func_clamp_shared_expert) was lost when themHC revert landed, so training clamped what DeepSeek-V4 and the engine leave unclamped.
csa.pyreadOptionaldsa_indexer_loss_coeffwithgetattr(..., 0.0)— whose default only fires when the attribute is missing — soNonereached
kl_div * loss_coeff. Two other sites in the same file already did it correctly.Miles side (this PR)
--dsv4-impl, with the arg translation and the three hard-errors, plus its unit test.impls; mbridge, the quantizers, the weight checker and the converters follow the rename.
mark_keep_in_fp32.disabled=in Bump Megatron-LM to miles-main-20260819 (latest NVIDIA dev) #2673).pipeline_model_parallel_layout.Verification (devbox, 8xH200)
4-layer RL e2e, default thresholds, no skip list, no relaxed gate — matches the pre-bump stack
(miles
49cac7b52+ megatron2e64d788b):train_rollout_logprob_abs_diffstep 0 / 1train_rollout_klstep 0 / 1found NaNBoth impls, same recorded rollout, same args, TP=1:
train_rollout_logprob_abs_diffNative mHC and miles mHC agree to 2.1e-3. Converting with
--dsv4-impl megatrongives27,390,526,557 parameters, identical to the miles impl.
Not covered: every sample in that rollout has reward 0, so GRPO advantage and
grad_normare 0on both impls — the backward ran clean but its numerics are untested. Also untested: TP>1 (rejected
by design), the cudnn kernel backend (needs
flash_mla, absent from the image), multi-step runs ofthe megatron impl.
Full investigation, including the wrong turns, in
bump_docs/05-dsv4-dual-backend.md.50-step A/B on 8×GB300 (32 GPU, sm103)
Identical config for both runs — TP1 · PP8 (first 4 / last 3 layers) · CP1 · EP4 · DP4,
max-tokens-per-gpu 4096, FP8 blockwise, fp32 optimizer, GRPO on DAPO-math-17k (32 prompts × 8
samples, max response 4096, lr 1e-6, deterministic mode), cudnn kernel backend for conversion and
training — only
--dsv4-impldiffers. megatron = job 2695 (50 steps), miles = job 2703 (54 steps).Timing, p50 over steps 2+ (steps 0–1 excluded: per-shape JIT warm-up). Full cycle = wall-clock
between consecutive rollout completions.
train_rollout_kl, mean