Skip to content

fix(megatron): mirror router GEMM precision onto bridge-built providers - #3263

Open
Arist12 wants to merge 5 commits into
radixark:mainfrom
Arist12:amd/bridge-router-gemm-fidelity
Open

Arist12 wants to merge 5 commits into
radixark:mainfrom
Arist12:amd/bridge-router-gemm-fidelity

Conversation

@Arist12

@Arist12 Arist12 commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Both bridge paths copy a whitelist of Megatron args onto the Megatron-Bridge provider.
moe_router_use_torch_mm is in neither list, so the provider keeps the TransformerConfig
default False — even though miles forces the arg on for every run:

# miles/backends/megatron_utils/arguments.py, set_default_megatron_args
# Before 20260819, radixark/Megatron-LM pick torch gemm for router, which is fp32 x fp32 ->
# fp32, and 20260819 convert the default to TE gemm which is bf16 x bf16 -> fp32. ...
args.moe_router_use_torch_mm = True

The provider is the model config, so TopKRouter reads the unset value and
RouterGatingLinearFunction takes the TE branch:

if te_general_gemm is not None and not use_torch_mm and router_dtype != torch.float64:
    output = te_general_gemm(weight, inp, router_dtype, layout="TN", bias=bias)  # bf16 operands
else:
    output = torch.mm(inp.to(router_dtype), weight.to(router_dtype).t())         # upcast first

te_general_gemm is available on ROCm, so on scripts/amd/run_gpt_oss_20b_lora.py the router
GEMM runs on bf16 operands while the resolved args table reports fp32 + torch mm.
moe_router_dtype is dropped by the non-LoRA whitelist in the same way. Building the provider
exactly as miles does:

field                        provider   requested
moe_router_use_torch_mm      False      True
moe_router_dtype             None       'fp32'

Megatron reports the same thing on every rank — the warning fires exactly on
num_moe_experts >= 32 and not self.moe_router_dtype, and gpt-oss-20b has 32 experts:

transformer_config.py:3597: UserWarning: Using a large number of experts (e.g. >=32) without
fp32 routing. Consider enabling moe_router_dtype for better numerical stability.

Change

Four assignments, at the existing MoE block of each whitelist.

moe_router_use_torch_mm is copied unconditionally: nothing in megatron.bridge ever assigns
it, so no provider has an opinion to preserve. moe_router_dtype is None-guarded because ~13
megatron.bridge providers deliberately default it to fp32, and an unconditional copy would
downgrade them.

Validation

8×MI355X gpt-oss-20b LoRA run, before/after: the warning goes 2 → 0 and the args table's values
are now the effective ones.

tests/fast/backends/megatron_utils/ — 326 passed, 2 skipped. The two runtime_args fixture
lines are required, not cosmetic: the new bare reads make the existing test_bridge_mtp_detachment
cases fail with AttributeError without them. Bare access matches the file's convention —
Megatron auto-registers every TransformerConfig field as a CLI arg, so both always exist on a
real parsed args; the neighbouring getattr guards cover miles-only or derived fields.

One misconfiguration becomes loud: --moe-router-dtype fp64 together with --moe-combine-in-fp32
now trips the assert at transformer_config.py:3402, where the bridge's fp32 default previously
won silently. No in-repo recipe is affected — every --moe-router-dtype under scripts/ and
examples/ passes fp32.

Supersedes #3257 (now closed), which covered only the LoRA-path moe_router_dtype line.

Rebase onto the multi-LoRA v2 move

[multi-lora v2 0/7] renamed bridge_lora_helpers.py to lora/bridge.py, which is what
turned this PR red. Rebased onto f6d0257; the conflict was purely positional, so the four
assignments are unchanged and land at the same MoE block in the new file. Re-ran
tests/fast/backends/megatron_utils/ on the rebase: 314 passed, 2 skipped (the count moved
with main, not with this change).

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@Arist12
Arist12 force-pushed the amd/bridge-router-gemm-fidelity branch from 015c1e2 to 72e533c Compare September 15, 2026 17:42
Both bridge whitelists drop moe_router_use_torch_mm, so the provider keeps the
TransformerConfig default False even though set_default_megatron_args forces it
True for every run, and the router GEMM takes the TE bf16 branch while the args
table reports torch mm. Mirror it on both paths, plus moe_router_dtype on the
non-LoRA path, guarded because bridge MoE providers default it to fp32.
@Arist12
Arist12 force-pushed the amd/bridge-router-gemm-fidelity branch from 72e533c to 719c53e Compare September 16, 2026 18:42
@XinyuJiangCMU

Copy link
Copy Markdown
Contributor

LGTM. This correctly passes the router settings to the Bridge provider. The LoRA changes overlap with the recently #3584

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants