Repository navigation
fix: stabilize MoE accumulation and support absorbed MLA adapters - #104
Closed
yueming-yuan wants to merge 8 commits into
Closed
yueming-yuan wants to merge 8 commits into
yueming-yuan wants to merge 8 commits into
Conversation
…#4452) Megatron-Bridge main (radixark bridge-new) imports megatron.core.transformer.mla_qk_norm_config.get_backend and overrides MLASelfAttention._resolve_qk_norm_config in its DeepSeek/GLM-5 attention (models/deepseek/attention.py); on miles-main the import fails and the hook does not exist, so every Bridge-backed path (LoRA, multi-LoRA, bridge-mode weight update) dies at import. Ported from upstream 4464d1c "Port Multi-Latent Attention to HybridModel (NVIDIA#4452)", MLA half only: * new megatron/core/transformer/mla_qk_norm_config.py (identical to upstream main @ f6c33bd) * MLASelfAttention / FusedMLASelfAttention build q/kv norms and up-projections through _resolve_qk_norm_config -> QKNormConfigResolver * AbsorbedMLA does the same; the combined K/V up-projection goes through the resolver, the miles-main split K/V path keeps its explicit submodules * backends.get_backend(transformer_impl) (Literal import) * the qk_layernorm test refactors that applied cleanly Not ported: the HybridModel MLA support (hybrid_block / hybrid_layer_allocation / hybrid_layer_specs, Symbols.MLA in arguments.py, hybrid tests) -- it conflicts with the miles-main hybrid code and nothing in miles uses HybridModel MLA. (cherry picked from commit 4464d1c, hybrid-model parts dropped) Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
…a_net Upstream Megatron-LM renamed the Gated DeltaNet variant to "gdn" (NVIDIA#5765, with "gated_delta_net" kept as a deprecated alias). Megatron-Bridge main selects Qwen3.5 linear attention with the new name, so on miles-main every Qwen3.5 model build died in get_experimental_attention_variant_module_spec with "Invalid experimental attention variant: gdn". miles-main keeps "gated_delta_net" as the canonical spelling and now treats "gdn" as an alias: * transformer_config: Literal accepts "gdn"; __post_init__ canonicalizes it; is_gated_delta_net_variant() replaces the literal comparisons so a value assigned after construction (Bridge sets it on the provider and never re-runs __post_init__) is recognized as well. * experimental_attention_variant_module_specs: dispatch and is_linear_attention_variant go through the helper, which is also exported from this module (Megatron-Bridge looks it up there). * training.py FLOP accounting accepts both spellings. "gdn2" (GDN2 kernels, upstream NVIDIA#5765) is intentionally not accepted: miles-main has no implementation for it and should keep failing loudly. Verified with Megatron-Bridge bridge-new: a GPTModelProvider configured like qwen35_bridge (experimental_attention_variant="gdn", linear_attention_freq=4, transformer_layer_spec=get_transformer_block_with_experimental_attention_variant_spec) raised the ValueError before and now resolves to GatedDeltaNet layers. Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
…for Megatron-Bridge Upstream Megatron-Core split Megatron-FSDP into FullyShardedDataParallelV1 (the implementation miles-main ships as FullyShardedDataParallel) and FullyShardedDataParallelV2, and turned FullyShardedDataParallel into a factory (NVIDIA#5865). Megatron-Bridge main imports both names from mcore_fsdp_adapter since NVIDIA-NeMo/Megatron-Bridge#5933 (conversion/utils.py::unwrap_model, training/setup.py, training/train.py), so on miles-main every bridge-mode weight load / export raised ImportError with bridge-new @ 2e09c234a. Add FullyShardedDataParallelV1 as an alias of the existing class and a FullyShardedDataParallelV2 placeholder whose constructor raises NotImplementedError. The v2 implementation is not ported and FullyShardedDataParallel stays the v1 class, so the existing isinstance checks in megatron/core/utils.py and megatron/training/training.py are unchanged. Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
Signed-off-by: yueming-yuan <yym022502@gmail.com>
yueming-yuan
marked this pull request as ready for review
September 22, 2026 21:21
Signed-off-by: yueming-yuan <yym022502@gmail.com>
Signed-off-by: yueming-yuan <yym022502@gmail.com>
Signed-off-by: yueming-yuan <yym022502@gmail.com>
7 of 15 tasks
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Includes the compatibility changes from #102, required by the Bridge version used in radixark/miles#3584. MoE combine:
593a05a37; dispatch backward:07fce1761; absorbed MLA:85e5de5e7.BF16/FP16 scatter-add rounds after every atomic addition, amplifying accumulation-order noise. Accumulate unfused unpermute in FP32 and cast once at the output. The fused TE path is unchanged; this also affects full-model unfused MoE execution and uses extra FP32 activation memory. Dispatch now promotes FP16/BF16 tensors requiring gradients around index_select and restores their dtype, so backward sums expert contributions in FP32. Forward-only, FP32 and FP64 dispatch keep their original computation.
Fixed/unfixed H200 probes cover FP16/BF16 with and without routing weights, 12 repeats each, FP64 reference error bounds, and exact input/probability gradients. Fixed outputs satisfy the FP32 accumulation plus final activation-rounding bound; the original implementation exceeds it. FP32 atomic addition is not guaranteed bitwise deterministic. Whole-model diagnostic interventions removed observed repeated-logprob drift in Qwen3-30B and GPT-OSS-20B/120B.
Absorbed MLA previously read the KV-up weight directly, failing on MultiLoRALinear (and a base-only weight property would silently omit adapters). Add optional token-dependent K/V projection hooks and let DSA return latent attention output; the caller applies the V projection. Ordinary weight projections retain their path. Adapter math lives in Bridge #41.
On the retained GLM-5.2 5-layer, 22.84B, TP4/SP/cuDNN model, exact changed methods unblock forward/backward and match zero/nonzero/disabled/bare-base references. Independent FP64 projection output/input/A/B gradient error is at most 8.53e-14. Dispatch validation covers FP16/BF16 × padded/unpadded × probabilities/no-probabilities on4 ranks,12 repeats: fixed gradient error against FP64 sum rounded to the activation dtype is0; original error reaches0.0009765625(BF16)/0.0001220703125(FP16). Forward outputs are unchanged. FP32 accumulation is not a bitwise-determinism guarantee.
Residual GLM repeated-backward variation is independently localized to cuDNN dKV FP32 atomics crossing BF16 rounding midpoints; same-input24-repeat dQ is exact. Against dense FP64, dQ/dKV relative L2 is0.468–0.834%/0.185–0.215%. Holding identical-input dKV fixed isolates recompute: core gradient difference0, up/both relative≤4.64e-6. No cuDNN numerical policy is changed here.
Transient dispatch cast storage is about27MiB for128×6144 tokens/top8, scaling to1.69GiB for8192 tokens. A fresh process with both dependency heads, optimizer-trained snapshots, long sparse sequences and full744B remain untested.
Changed-file Pylint/isort and Black26 pass. Full Megatron pre-commit remains blocked by existing unrelated Pylint failures and a Black24.4.2 hook versus required26 mismatch. Dependency for radixark/miles#3584.