Skip to content

[megatron] Fix the distributed optimizer for param groups mixing fp32 and bf16 - #2539

Merged
avigyabb merged 4 commits into
NovaSky-AI:mainfrom
avigyabb:pr/megatron-mixed-dtype-params
Oct 27, 2026
Merged

avigyabb merged 4 commits into
NovaSky-AI:mainfrom
avigyabb:pr/megatron-mixed-dtype-params

Conversation

@avigyabb

@avigyabb avigyabb commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Models that train some params in fp32 under bf16 (mHC mixing weights in DeepSeek-V4 / V4.1 and GLM-5.3, DeepSeek-V4's compressor ape) put both dtypes in one optimizer param group. Two megatron-core paths mishandle it:

  • Checkpoint index. DistributedOptimizer records each model param's position in grad-buffer order (bf16, then fp32) but sets the group's params to [fp32 shards..., bf16 shards...], so every model param resolves to the next one's main param and optimizer state. Only checkpoint save/load read the index; full-FT save failed in sharded_param_state_* with a shape mismatch (the GLM-5.3 full-FT checkpoint row's documented failure). patch_distopt_mixed_dtype_group_index rebuilds the index after init.

  • Grad norm / clip with the precision-aware optimizer. The norm and clip lists are then the decoupled_grad views, which keep each param's dtype, and one TransformerEngine multi-tensor kernel call dispatches on the first tensor's dtype and reads past the bf16 tensors: the first optimizer step dies with "CUDA error: an illegal memory access" in multi_tensor_apply. patch_clip_grads_mixed_dtype splits those calls by dtype.

Unit tests cover the rebuilt index and the dtype split.

Testing

  • CPU: tests/backends/skyrl_train/patches/megatron/test_distopt_mixed_dtype_group_index.py and test_clip_grads_mixed_dtype.py pass.
  • End to end: DeepSeek-V4-Flash / V4.1-Flash full-FT checkpoint save/resume rows (test_trainer_full_checkpointing.py) pass with the index fix. The illegal memory access reproduced on a 1-node V4.1 slice run (precision-aware optimizer + CPU offload, colocated) and is gone with the dtype split; the 6-node V4.1 full-FT runs then trained 10 steps.

Split out of the DeepSeek-V4-Flash / V4.1-Flash support work; independent of the other PRs in that series. GPU rows have not been re-run on this exact branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC


Note

Medium Risk
Touches distributed optimizer checkpoint indexing and global grad-norm/clip wrapping on every Megatron training step; limited to mixed-dtype param groups but errors would affect checkpoint integrity or training stability.

Overview
Adds two temporary megatron-core patches for models that keep some trainable params in fp32 under bf16 (e.g. mHC on DeepSeek-V4/V4.1 and GLM-5.3), where a single optimizer group mixes dtypes.

Distributed optimizer checkpointing: After DistributedOptimizer.__init__, rebuilds model_param_group_index_map so each model param maps to the correct main param (fp32 shards first, then bf16-derived), fixing full-FT checkpoint save/load shape mismatches in sharded_param_state_* (upstream NVIDIA/Megatron-LM#2872).

Grad norm / clip: Wraps clip_grads.multi_tensor_applier to run once per dtype when the precision-aware optimizer passes mixed bf16/fp32 decoupled_grad lists, then recombines L2 norms; avoids illegal memory access on the first step (upstream #7279).

Both patches are applied at megatron_worker import, documented in patches/megatron/README.md with landed checks and removal steps, with CPU tests for index rebuild, dtype splitting, and wrapper behavior.

Reviewed by Cursor Bugbot for commit 535a0fa. Bugbot is set up for automated code reviews on this repo. Configure here.

… and bf16

Models that train some params in fp32 under bf16 (mHC mixing weights in
DeepSeek-V4 / V4.1 and GLM-5.3, DeepSeek-V4's compressor ape) put both
dtypes in one optimizer param group. Two megatron-core paths mishandle it:

- Checkpoint index. DistributedOptimizer records each model param's position
  in grad-buffer order (bf16, then fp32) but sets the group's params to
  [fp32 shards..., bf16 shards...], so every model param resolves to the
  next one's main param and optimizer state. Only checkpoint save/load read
  the index; full-FT save failed in sharded_param_state_* with a shape
  mismatch (the GLM-5.3 full-FT checkpoint row's documented failure).
  patch_distopt_mixed_dtype_group_index rebuilds the index after init.

- Grad norm / clip with the precision-aware optimizer. The norm and clip
  lists are then the decoupled_grad views, which keep each param's dtype,
  and one TransformerEngine multi-tensor kernel call dispatches on the first
  tensor's dtype and reads past the bf16 tensors: the first optimizer step
  dies with "CUDA error: an illegal memory access" in multi_tensor_apply.
  patch_clip_grads_mixed_dtype splits those calls by dtype.

Unit tests cover the rebuilt index and the dtype split.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces two patches for Megatron-Core to support mixed-precision (fp32 and bf16) parameter groups, which are used in models like DeepSeek-V4 and GLM-5.3. The first patch splits mixed-dtype gradient lists before calling multi-tensor kernels to prevent CUDA errors, and the second patch rebuilds the distributed optimizer's model-to-main parameter index map to avoid shape mismatches during checkpoint saving. The review feedback recommends enhancing the robustness of these patches by validating group lengths before zipping them and using safe attribute retrieval (getattr) to prevent potential AttributeError exceptions if Megatron-Core attributes are missing or renamed.

Comment on lines +27 to +30
def rebuild_model_param_group_index_map(model_fp32_groups, model_float16_groups) -> dict:
"""Index each model param by its main param's position: the group's fp32 params, then its fp16/bf16 ones."""
index_map = {}
for group_index, (fp32_params, float16_params) in enumerate(zip(model_fp32_groups, model_float16_groups)):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent silent truncation and potential parameter index mapping corruption, we should verify that model_fp32_groups and model_float16_groups have the same length before zipping them.

Suggested change
def rebuild_model_param_group_index_map(model_fp32_groups, model_float16_groups) -> dict:
"""Index each model param by its main param's position: the group's fp32 params, then its fp16/bf16 ones."""
index_map = {}
for group_index, (fp32_params, float16_params) in enumerate(zip(model_fp32_groups, model_float16_groups)):
def rebuild_model_param_group_index_map(model_fp32_groups, model_float16_groups) -> dict:
"""Index each model param by its main param's position: the group's fp32 params, then its fp16/bf16 ones."""
if len(model_fp32_groups) != len(model_float16_groups):
raise ValueError(f"Mismatched group lengths: {len(model_fp32_groups)} vs {len(model_float16_groups)}")
index_map = {}
for group_index, (fp32_params, float16_params) in enumerate(zip(model_fp32_groups, model_float16_groups)):

Comment on lines +57 to +61
rebuilt = rebuild_model_param_group_index_map(fp32_groups, float16_groups)
if rebuilt != self.model_param_group_index_map:
moved = sum(1 for p, v in rebuilt.items() if self.model_param_group_index_map.get(p) != v)
logger.info(f"Distributed optimizer: re-indexed {moved} params in mixed fp32/bf16 param groups")
self.model_param_group_index_map = rebuilt

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If self.model_param_group_index_map is not initialized or has been renamed/removed in the Megatron-Core version being used, accessing it directly will raise an AttributeError. We should safely retrieve it using getattr to ensure robustness.

Suggested change
rebuilt = rebuild_model_param_group_index_map(fp32_groups, float16_groups)
if rebuilt != self.model_param_group_index_map:
moved = sum(1 for p, v in rebuilt.items() if self.model_param_group_index_map.get(p) != v)
logger.info(f"Distributed optimizer: re-indexed {moved} params in mixed fp32/bf16 param groups")
self.model_param_group_index_map = rebuilt
rebuilt = rebuild_model_param_group_index_map(fp32_groups, float16_groups)
current_map = getattr(self, "model_param_group_index_map", None)
if current_map is not None and rebuilt != current_map:
moved = sum(1 for p, v in rebuilt.items() if current_map.get(p) != v)
logger.info(f"Distributed optimizer: re-indexed {moved} params in mixed fp32/bf16 param groups")
self.model_param_group_index_map = rebuilt

Comment on lines +33 to +40
try:
import torch
from megatron.core.optimizer import clip_grads
except ImportError:
return False

inner = clip_grads.multi_tensor_applier
l2_norm_impl = getattr(clip_grads, "l2_norm_impl", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential AttributeError if clip_grads does not have multi_tensor_applier (e.g., due to changes in Megatron-Core versions), we should safely retrieve it using getattr inside the try-except block.

Suggested change
try:
import torch
from megatron.core.optimizer import clip_grads
except ImportError:
return False
inner = clip_grads.multi_tensor_applier
l2_norm_impl = getattr(clip_grads, "l2_norm_impl", None)
try:
import torch
from megatron.core.optimizer import clip_grads
inner = getattr(clip_grads, "multi_tensor_applier", None)
if inner is None:
return False
except ImportError:
return False
l2_norm_impl = getattr(clip_grads, "l2_norm_impl", None)

@greptile-apps

greptile-apps Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The changes affect mixed-dtype Megatron training and its checkpoint workflow.

The PR appears safe to merge, with a non-blocking request to test the gradient wrapper directly.

Findings

  1. P2 Gradient wrapper remains untested ▶

Reviews (1) · Last reviewed commit: "[megatron] Fix the distributed optimizer..." · Reviewed by Greptile

avigyabb and others added 2 commits October 27, 2026 21:14
The tests only exercised split_by_dtype. With a fake one-dtype-per-call
kernel: a mixed L2 norm is recombined as sqrt(sum of squared group norms),
a mixed scale clips both dtype groups, a uniform list passes through
unchanged, and applying the patch twice wraps once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
…oval-plan entries

patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main)
does the same index rebuild; #5835 merged it into dev only, so it is not in
the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open;
issue #7338) splits by dtype inside get_grad_norm_fp32 /
clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and
at the call sites; README sections with the landed check and what to remove.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
@avigyabb

Copy link
Copy Markdown
Collaborator Author

Upstream fixes and removal plan for the two patches in this PR (both stay needed with SkyRL's current megatron-core pin):

Patch Upstream Status Shape
patch_distopt_mixed_dtype_group_index.py NVIDIA/Megatron-LM#2872 open, main Same fix: rebuild model_param_group_index_map as [*model_fp32_params, *model_float16_params] per group at the end of DistributedOptimizer.__init__.
NVIDIA/Megatron-LM#5835 merged 2026-07-18 into dev only Same rebuild. Its merge commit is not an ancestor of the pin (07147d69, main: diverged, 886 ahead / 552 behind), so it does not help yet. It (and #2872) also adds a no_grad / detach for the non-precision-aware fp32 path, which the SkyRL recipes don't take.
patch_clip_grads_mixed_dtype.py NVIDIA/Megatron-LM#7279 (issue NVIDIA/Megatron-LM#7338) open, main Same math: split by dtype and sum the squared norms. Upstream does it inside get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 (_group_grads_by_dtype); this patch wraps the one multi_tensor_applier they both call, so it doesn't vendor either function.

Pushed 333df1e: TODOs pointing at these PRs in both patch modules and at their call sites in megatron_worker.py, plus patches/megatron/README.md removal-plan sections (landed check and what to delete) and test rows for both. When a megatron-core bump includes #2872 (or #5835 reaches main) and #7279, remove the matching patch, its call, and its CPU test.

89f8f92 also adds CPU tests for the multi_tensor_applier wrapper itself (mixed L2 norm recombined, both dtype groups clipped, uniform lists passed through, applied once).

…pe-params

# Conflicts:
#	skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py
@avigyabb
avigyabb merged commit 0d529c9 into NovaSky-AI:main Oct 27, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant