Skip to content

fix: stabilize MoE accumulation and support absorbed MLA adapters - #104

Closed
yueming-yuan wants to merge 8 commits into
radixark:miles-mainfrom
yueming-yuan:yueming/multilora-combine
Closed

yueming-yuan wants to merge 8 commits into
radixark:miles-mainfrom
yueming-yuan:yueming/multilora-combine

Conversation

@yueming-yuan

@yueming-yuan yueming-yuan commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Includes the compatibility changes from #102, required by the Bridge version used in radixark/miles#3584. MoE combine: 593a05a37; dispatch backward: 07fce1761; absorbed MLA: 85e5de5e7.

BF16/FP16 scatter-add rounds after every atomic addition, amplifying accumulation-order noise. Accumulate unfused unpermute in FP32 and cast once at the output. The fused TE path is unchanged; this also affects full-model unfused MoE execution and uses extra FP32 activation memory. Dispatch now promotes FP16/BF16 tensors requiring gradients around index_select and restores their dtype, so backward sums expert contributions in FP32. Forward-only, FP32 and FP64 dispatch keep their original computation.

Fixed/unfixed H200 probes cover FP16/BF16 with and without routing weights, 12 repeats each, FP64 reference error bounds, and exact input/probability gradients. Fixed outputs satisfy the FP32 accumulation plus final activation-rounding bound; the original implementation exceeds it. FP32 atomic addition is not guaranteed bitwise deterministic. Whole-model diagnostic interventions removed observed repeated-logprob drift in Qwen3-30B and GPT-OSS-20B/120B.

Absorbed MLA previously read the KV-up weight directly, failing on MultiLoRALinear (and a base-only weight property would silently omit adapters). Add optional token-dependent K/V projection hooks and let DSA return latent attention output; the caller applies the V projection. Ordinary weight projections retain their path. Adapter math lives in Bridge #41.

On the retained GLM-5.2 5-layer, 22.84B, TP4/SP/cuDNN model, exact changed methods unblock forward/backward and match zero/nonzero/disabled/bare-base references. Independent FP64 projection output/input/A/B gradient error is at most 8.53e-14. Dispatch validation covers FP16/BF16 × padded/unpadded × probabilities/no-probabilities on4 ranks,12 repeats: fixed gradient error against FP64 sum rounded to the activation dtype is0; original error reaches0.0009765625(BF16)/0.0001220703125(FP16). Forward outputs are unchanged. FP32 accumulation is not a bitwise-determinism guarantee.

Residual GLM repeated-backward variation is independently localized to cuDNN dKV FP32 atomics crossing BF16 rounding midpoints; same-input24-repeat dQ is exact. Against dense FP64, dQ/dKV relative L2 is0.468–0.834%/0.185–0.215%. Holding identical-input dKV fixed isolates recompute: core gradient difference0, up/both relative≤4.64e-6. No cuDNN numerical policy is changed here.

Transient dispatch cast storage is about27MiB for128×6144 tokens/top8, scaling to1.69GiB for8192 tokens. A fresh process with both dependency heads, optimizer-trained snapshots, long sparse sequences and full744B remain untested.

Changed-file Pylint/isort and Black26 pass. Full Megatron pre-commit remains blocked by existing unrelated Pylint failures and a Black24.4.2 hook versus required26 mismatch. Dependency for radixark/miles#3584.

yushengsu-thu and others added 5 commits September 16, 2026 22:56
…#4452)

Megatron-Bridge main (radixark bridge-new) imports
megatron.core.transformer.mla_qk_norm_config.get_backend and overrides
MLASelfAttention._resolve_qk_norm_config in its DeepSeek/GLM-5 attention
(models/deepseek/attention.py); on miles-main the import fails and the hook
does not exist, so every Bridge-backed path (LoRA, multi-LoRA, bridge-mode
weight update) dies at import.

Ported from upstream 4464d1c "Port Multi-Latent Attention to HybridModel
(NVIDIA#4452)", MLA half only:

* new megatron/core/transformer/mla_qk_norm_config.py (identical to upstream
  main @ f6c33bd)
* MLASelfAttention / FusedMLASelfAttention build q/kv norms and up-projections
  through _resolve_qk_norm_config -> QKNormConfigResolver
* AbsorbedMLA does the same; the combined K/V up-projection goes through the
  resolver, the miles-main split K/V path keeps its explicit submodules
* backends.get_backend(transformer_impl) (Literal import)
* the qk_layernorm test refactors that applied cleanly

Not ported: the HybridModel MLA support (hybrid_block / hybrid_layer_allocation /
hybrid_layer_specs, Symbols.MLA in arguments.py, hybrid tests) -- it conflicts
with the miles-main hybrid code and nothing in miles uses HybridModel MLA.

(cherry picked from commit 4464d1c, hybrid-model parts dropped)

Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
…a_net

Upstream Megatron-LM renamed the Gated DeltaNet variant to "gdn" (NVIDIA#5765, with
"gated_delta_net" kept as a deprecated alias). Megatron-Bridge main selects
Qwen3.5 linear attention with the new name, so on miles-main every Qwen3.5
model build died in get_experimental_attention_variant_module_spec with
"Invalid experimental attention variant: gdn".

miles-main keeps "gated_delta_net" as the canonical spelling and now treats
"gdn" as an alias:

* transformer_config: Literal accepts "gdn"; __post_init__ canonicalizes it;
  is_gated_delta_net_variant() replaces the literal comparisons so a value
  assigned after construction (Bridge sets it on the provider and never
  re-runs __post_init__) is recognized as well.
* experimental_attention_variant_module_specs: dispatch and
  is_linear_attention_variant go through the helper, which is also exported
  from this module (Megatron-Bridge looks it up there).
* training.py FLOP accounting accepts both spellings.

"gdn2" (GDN2 kernels, upstream NVIDIA#5765) is intentionally not accepted: miles-main
has no implementation for it and should keep failing loudly.

Verified with Megatron-Bridge bridge-new: a GPTModelProvider configured like
qwen35_bridge (experimental_attention_variant="gdn", linear_attention_freq=4,
transformer_layer_spec=get_transformer_block_with_experimental_attention_variant_spec)
raised the ValueError before and now resolves to GatedDeltaNet layers.

Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
…for Megatron-Bridge

Upstream Megatron-Core split Megatron-FSDP into FullyShardedDataParallelV1 (the
implementation miles-main ships as FullyShardedDataParallel) and
FullyShardedDataParallelV2, and turned FullyShardedDataParallel into a factory
(NVIDIA#5865). Megatron-Bridge main imports both names from
mcore_fsdp_adapter since NVIDIA-NeMo/Megatron-Bridge#5933
(conversion/utils.py::unwrap_model, training/setup.py, training/train.py), so on
miles-main every bridge-mode weight load / export raised ImportError with
bridge-new @ 2e09c234a.

Add FullyShardedDataParallelV1 as an alias of the existing class and a
FullyShardedDataParallelV2 placeholder whose constructor raises
NotImplementedError. The v2 implementation is not ported and
FullyShardedDataParallel stays the v1 class, so the existing isinstance checks
in megatron/core/utils.py and megatron/training/training.py are unchanged.

Signed-off-by: Yusheng Su <yushengsu.thu@gmail.com>
Signed-off-by: yueming-yuan <yym022502@gmail.com>
Signed-off-by: yueming-yuan <yym022502@gmail.com>
Signed-off-by: yueming-yuan <yym022502@gmail.com>
@yueming-yuan yueming-yuan changed the title fix: accumulate unfused MoE outputs in FP32 fix: stabilize MoE combine and support absorbed MLA adapters Sep 23, 2026
Signed-off-by: yueming-yuan <yym022502@gmail.com>
@yueming-yuan yueming-yuan changed the title fix: stabilize MoE combine and support absorbed MLA adapters fix: stabilize MoE accumulation and support absorbed MLA adapters Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants