Repository navigation
[megatron] Fix the distributed optimizer for param groups mixing fp32 and bf16 - #2539
Conversation
… and bf16 Models that train some params in fp32 under bf16 (mHC mixing weights in DeepSeek-V4 / V4.1 and GLM-5.3, DeepSeek-V4's compressor ape) put both dtypes in one optimizer param group. Two megatron-core paths mishandle it: - Checkpoint index. DistributedOptimizer records each model param's position in grad-buffer order (bf16, then fp32) but sets the group's params to [fp32 shards..., bf16 shards...], so every model param resolves to the next one's main param and optimizer state. Only checkpoint save/load read the index; full-FT save failed in sharded_param_state_* with a shape mismatch (the GLM-5.3 full-FT checkpoint row's documented failure). patch_distopt_mixed_dtype_group_index rebuilds the index after init. - Grad norm / clip with the precision-aware optimizer. The norm and clip lists are then the decoupled_grad views, which keep each param's dtype, and one TransformerEngine multi-tensor kernel call dispatches on the first tensor's dtype and reads past the bf16 tensors: the first optimizer step dies with "CUDA error: an illegal memory access" in multi_tensor_apply. patch_clip_grads_mixed_dtype splits those calls by dtype. Unit tests cover the rebuilt index and the dtype split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
There was a problem hiding this comment.
Code Review
This pull request introduces two patches for Megatron-Core to support mixed-precision (fp32 and bf16) parameter groups, which are used in models like DeepSeek-V4 and GLM-5.3. The first patch splits mixed-dtype gradient lists before calling multi-tensor kernels to prevent CUDA errors, and the second patch rebuilds the distributed optimizer's model-to-main parameter index map to avoid shape mismatches during checkpoint saving. The review feedback recommends enhancing the robustness of these patches by validating group lengths before zipping them and using safe attribute retrieval (getattr) to prevent potential AttributeError exceptions if Megatron-Core attributes are missing or renamed.
| def rebuild_model_param_group_index_map(model_fp32_groups, model_float16_groups) -> dict: | ||
| """Index each model param by its main param's position: the group's fp32 params, then its fp16/bf16 ones.""" | ||
| index_map = {} | ||
| for group_index, (fp32_params, float16_params) in enumerate(zip(model_fp32_groups, model_float16_groups)): |
There was a problem hiding this comment.
To prevent silent truncation and potential parameter index mapping corruption, we should verify that model_fp32_groups and model_float16_groups have the same length before zipping them.
| def rebuild_model_param_group_index_map(model_fp32_groups, model_float16_groups) -> dict: | |
| """Index each model param by its main param's position: the group's fp32 params, then its fp16/bf16 ones.""" | |
| index_map = {} | |
| for group_index, (fp32_params, float16_params) in enumerate(zip(model_fp32_groups, model_float16_groups)): | |
| def rebuild_model_param_group_index_map(model_fp32_groups, model_float16_groups) -> dict: | |
| """Index each model param by its main param's position: the group's fp32 params, then its fp16/bf16 ones.""" | |
| if len(model_fp32_groups) != len(model_float16_groups): | |
| raise ValueError(f"Mismatched group lengths: {len(model_fp32_groups)} vs {len(model_float16_groups)}") | |
| index_map = {} | |
| for group_index, (fp32_params, float16_params) in enumerate(zip(model_fp32_groups, model_float16_groups)): |
| rebuilt = rebuild_model_param_group_index_map(fp32_groups, float16_groups) | ||
| if rebuilt != self.model_param_group_index_map: | ||
| moved = sum(1 for p, v in rebuilt.items() if self.model_param_group_index_map.get(p) != v) | ||
| logger.info(f"Distributed optimizer: re-indexed {moved} params in mixed fp32/bf16 param groups") | ||
| self.model_param_group_index_map = rebuilt |
There was a problem hiding this comment.
If self.model_param_group_index_map is not initialized or has been renamed/removed in the Megatron-Core version being used, accessing it directly will raise an AttributeError. We should safely retrieve it using getattr to ensure robustness.
| rebuilt = rebuild_model_param_group_index_map(fp32_groups, float16_groups) | |
| if rebuilt != self.model_param_group_index_map: | |
| moved = sum(1 for p, v in rebuilt.items() if self.model_param_group_index_map.get(p) != v) | |
| logger.info(f"Distributed optimizer: re-indexed {moved} params in mixed fp32/bf16 param groups") | |
| self.model_param_group_index_map = rebuilt | |
| rebuilt = rebuild_model_param_group_index_map(fp32_groups, float16_groups) | |
| current_map = getattr(self, "model_param_group_index_map", None) | |
| if current_map is not None and rebuilt != current_map: | |
| moved = sum(1 for p, v in rebuilt.items() if current_map.get(p) != v) | |
| logger.info(f"Distributed optimizer: re-indexed {moved} params in mixed fp32/bf16 param groups") | |
| self.model_param_group_index_map = rebuilt |
| try: | ||
| import torch | ||
| from megatron.core.optimizer import clip_grads | ||
| except ImportError: | ||
| return False | ||
|
|
||
| inner = clip_grads.multi_tensor_applier | ||
| l2_norm_impl = getattr(clip_grads, "l2_norm_impl", None) |
There was a problem hiding this comment.
To prevent potential AttributeError if clip_grads does not have multi_tensor_applier (e.g., due to changes in Megatron-Core versions), we should safely retrieve it using getattr inside the try-except block.
| try: | |
| import torch | |
| from megatron.core.optimizer import clip_grads | |
| except ImportError: | |
| return False | |
| inner = clip_grads.multi_tensor_applier | |
| l2_norm_impl = getattr(clip_grads, "l2_norm_impl", None) | |
| try: | |
| import torch | |
| from megatron.core.optimizer import clip_grads | |
| inner = getattr(clip_grads, "multi_tensor_applier", None) | |
| if inner is None: | |
| return False | |
| except ImportError: | |
| return False | |
| l2_norm_impl = getattr(clip_grads, "l2_norm_impl", None) |
[Medium impact] The changes affect mixed-dtype Megatron training and its checkpoint workflow. The PR appears safe to merge, with a non-blocking request to test the gradient wrapper directly. Findings
Reviews (1) · Last reviewed commit: "[megatron] Fix the distributed optimizer..." · Reviewed by Greptile |
The tests only exercised split_by_dtype. With a fake one-dtype-per-call kernel: a mixed L2 norm is recombined as sqrt(sum of squared group norms), a mixed scale clips both dtype groups, a uniform list passes through unchanged, and applying the patch twice wraps once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
…oval-plan entries patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main) does the same index rebuild; #5835 merged it into dev only, so it is not in the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open; issue #7338) splits by dtype inside get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and at the call sites; README sections with the landed check and what to remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
|
Upstream fixes and removal plan for the two patches in this PR (both stay needed with SkyRL's current megatron-core pin):
Pushed 333df1e: TODOs pointing at these PRs in both patch modules and at their call sites in 89f8f92 also adds CPU tests for the |
…pe-params # Conflicts: # skyrl/backends/skyrl_train/workers/megatron/megatron_worker.py
Summary
Models that train some params in fp32 under bf16 (mHC mixing weights in DeepSeek-V4 / V4.1 and GLM-5.3, DeepSeek-V4's compressor ape) put both dtypes in one optimizer param group. Two megatron-core paths mishandle it:
Checkpoint index. DistributedOptimizer records each model param's position in grad-buffer order (bf16, then fp32) but sets the group's params to [fp32 shards..., bf16 shards...], so every model param resolves to the next one's main param and optimizer state. Only checkpoint save/load read the index; full-FT save failed in sharded_param_state_* with a shape mismatch (the GLM-5.3 full-FT checkpoint row's documented failure). patch_distopt_mixed_dtype_group_index rebuilds the index after init.
Grad norm / clip with the precision-aware optimizer. The norm and clip lists are then the decoupled_grad views, which keep each param's dtype, and one TransformerEngine multi-tensor kernel call dispatches on the first tensor's dtype and reads past the bf16 tensors: the first optimizer step dies with "CUDA error: an illegal memory access" in multi_tensor_apply. patch_clip_grads_mixed_dtype splits those calls by dtype.
Unit tests cover the rebuilt index and the dtype split.
Testing
tests/backends/skyrl_train/patches/megatron/test_distopt_mixed_dtype_group_index.pyandtest_clip_grads_mixed_dtype.pypass.test_trainer_full_checkpointing.py) pass with the index fix. The illegal memory access reproduced on a 1-node V4.1 slice run (precision-aware optimizer + CPU offload, colocated) and is gone with the dtype split; the 6-node V4.1 full-FT runs then trained 10 steps.Split out of the DeepSeek-V4-Flash / V4.1-Flash support work; independent of the other PRs in that series. GPU rows have not been re-run on this exact branch.
🤖 Generated with Claude Code
https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
Note
Medium Risk
Touches distributed optimizer checkpoint indexing and global grad-norm/clip wrapping on every Megatron training step; limited to mixed-dtype param groups but errors would affect checkpoint integrity or training stability.
Overview
Adds two temporary megatron-core patches for models that keep some trainable params in fp32 under bf16 (e.g. mHC on DeepSeek-V4/V4.1 and GLM-5.3), where a single optimizer group mixes dtypes.
Distributed optimizer checkpointing: After
DistributedOptimizer.__init__, rebuildsmodel_param_group_index_mapso each model param maps to the correct main param (fp32shards first, then bf16-derived), fixing full-FT checkpoint save/load shape mismatches insharded_param_state_*(upstream NVIDIA/Megatron-LM#2872).Grad norm / clip: Wraps
clip_grads.multi_tensor_applierto run once per dtype when the precision-aware optimizer passes mixed bf16/fp32decoupled_gradlists, then recombines L2 norms; avoids illegal memory access on the first step (upstream #7279).Both patches are applied at
megatron_workerimport, documented inpatches/megatron/README.mdwith landed checks and removal steps, with CPU tests for index rebuild, dtype splitting, and wrapper behavior.Reviewed by Cursor Bugbot for commit 535a0fa. Bugbot is set up for automated code reviews on this repo. Configure here.