Repository navigation
Fix mixed-dtype gradient norms and clipping - #7279
aroshanghias-nvd wants to merge 3 commits into
Conversation
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
|
Linking the community report in #7338: it describes the same mixed BF16/FP32 norm/clipping caller issue and now includes a single-GPU reproducer plus an H100 training case. This PR appears to cover the proposed caller-side correction, including both clipping-scale paths. @aroshanghias-nvd, please link that report in the PR description and let us know what remains before this is ready for review. The reporter's configuration would provide useful validation alongside the component tests already described here. |
|
Author review is complete, and I’ve linked #7338. The component and determinism tests pass as described. Validation of commit |
|
@wujingyue, could you review the mixed-dtype norm and clipping changes now that this is ready? The reporter has added a successful three-step H100 training smoke in #7338 (comment). They adapted the caller patch from 333ecc9 to their older dev checkout (43e45e1), so this supports the fix in their configuration rather than validating the complete PR checkout at its current head. |
| l2_norm_impl, dummy_overflow_buf, [dtype_grads], False | ||
| ) | ||
| # Combine local squared norms before the existing collectives. | ||
| if index == 0: |
There was a problem hiding this comment.
Can we simply initialize total_norm to 0.0 and get rid of this condition?
There was a problem hiding this comment.
Simplified to accumulate into the existing zero-valued CUDA tensor and removed the first-group conditional. Keeping the tensor initialization, rather than a Python float, also handles ranks with no local gradients when they reach the existing reductions. The reductions remain outside the grouping loop.
| total_norm = grad_norm**norm_type | ||
| for index, dtype_grads in enumerate(_group_grads_by_dtype(grads_for_norm)): | ||
| grad_norm, _ = multi_tensor_applier( | ||
| l2_norm_impl, dummy_overflow_buf, [dtype_grads], False |
There was a problem hiding this comment.
It's hard for readers to figure out what False mean. Consider a kwarg.
There was a problem hiding this comment.
Added # no per-tensor norms to explain the flag. I kept it positional because the supported local multi_tensor_applier wrapper accepts *args, not keyword arguments (megatron/core/utils.py).
There was a problem hiding this comment.
multi_tensor_applier wrapper accepts *args, not keyword arguments (megatron/core/utils.py).
Got it -- that's a bummer but worth a separate PR :)
| notes="multi_tensor l2norm / scale (TE, apex or local fallback), including mixed-dtype " | ||
| "caller bucketing and padded-storage replay, and fused Adam; " |
There was a problem hiding this comment.
The wording was intended to document the added mixed-dtype replay coverage. The existing optimizer entry already points to that test file, so I removed the redundant manifest change. The coverage checker still passes with the new replay tests.
There was a problem hiding this comment.
The test looks pretty verbose. For coverage, I'd just test one config that triggers mixed grad_dtype.
There was a problem hiding this comment.
Reduced the new numerical test matrix from 102 cases to 14, using one gradient length and the precision-aware decoupled-gradient path. I retained both dtype orders because they exercise different byte-width failures, and both coefficient APIs because norm correctness alone does not guarantee correct scaling. The small norm layout matrix retains empty and disjoint rank ownership coverage. Ordinary .grad remains exercised by the mixed-dtype determinism tests.
|
|
||
| @pytest.mark.parametrize('use_decoupled_grad', [False, True]) | ||
| @pytest.mark.parametrize('tensor_coefficient', [False, True]) | ||
| def test_empty_gradients_do_not_launch_kernel( |
There was a problem hiding this comment.
This is arguably testing implementation not behavior. It makes sense to verify grad clipping doesn't crash with empty gradients, but whether a kernel gets launched behind the scene is an implementation detail that I'd rather not test.
There was a problem hiding this comment.
Agreed. Replaced the kernel-launch assertion with a direct empty-gradient clipping call and CUDA synchronization, so the test checks successful completion rather than how the operation is implemented.
|
Again, this PR should go to @NVIDIA/mcore-optimizer. I happened to touch the optimizer code recently for FSDP, but I don't own the code. |
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
| assert num_zeros == grad.numel() - 1 | ||
|
|
||
|
|
||
| _DTYPES = [(torch.bfloat16, torch.float32), (torch.float32, torch.bfloat16)] |
There was a problem hiding this comment.
This is hard to read. The name doesn't say why the dtypes are in tuples.
Also, it's on the boundary of https://testing.googleblog.com/2019/12/testing-on-toilet-tests-too-dry-make.html. Consider letting each test manages its own parametrization with a well-documented intention.
There was a problem hiding this comment.
Inlined the dtype tuples in each test and documented the intention: fused kernels dispatch using the first tensor's dtype, so both mixed orders exercise distinct failure modes.
| grads, grad_stats_parallel_group=torch.distributed.group.WORLD | ||
| ) | ||
|
|
||
| assert float(actual) == pytest.approx(float(squared_norm.sqrt()), rel=2e-6, abs=1e-8) |
There was a problem hiding this comment.
torch.testing.assert_close instead of reinventing another way to compare?
There was a problem hiding this comment.
Replaced the norm comparison with torch.testing.assert_close, retaining the existing relative and absolute tolerances.
|
|
||
| @pytest.mark.parametrize('tensor_coefficient', [False, True]) | ||
| def test_clip_empty_gradients(self, tensor_coefficient): | ||
| if tensor_coefficient and clip_grads.multi_tensor_scale_tensor_impl is None: |
There was a problem hiding this comment.
| if tensor_coefficient and clip_grads.multi_tensor_scale_tensor_impl is None: |
IIUC, CI installs TE so this check is not so relevant. It's like we could ask every test to skipif not torch.cuda.is_available but nobody really cares :)
There was a problem hiding this comment.
Removed the backend skip checks from the mixed-dtype tests, including both tensor-coefficient checks. CI supplies Transformer Engine, so missing backend support should surface as a test failure.
Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
|
Addressed the remaining test feedback: each test now documents its dtype-order parameters, the norm assertion uses @wujingyue, could you refresh your approval after checking this cleanup? @NVIDIA/mcore-optimizer, could an optimizer owner review the caller fix so this can advance through the owner and final-review gates? |
|
Assigned the PR to @Connor-XY, who was the on-call when the PR was initially sent out. |
No need to -- approval sticks. I assigned this to @Connor-XY who'll help you get other approvals. |
## Summary - Add the SA-V tracking verifier and its server configuration. - Declare the SA-V environment servers used by video RL. - Add portable five-row example data, schema, metrics, and checked rollout artifacts. ## Why This is the Gym-side verifier foundation for mixed SA-V tracking and CapRL video training in NeMo-RL. No separate issue is needed; this is part of the Super 3.5 video RL upstreaming stack. ## Validation - Runtime evidence: the [healthy 16-node async `r18` W&B run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003) shows repeated GRPO cycles, stable TMPE, and increasing reward on the equivalent video implementation. - `gym dataset collate ... +mode=example_validation`: 5 examples; generated `data/example_metrics.json`. - Focused validation: 25 verifier tests and all 5 example rows against `task_data.py` passed. - Ruff format and lint passed for the changed Python files. - Both Gym stack ranges pass `git diff --check`. ## Rollouts `data/example_rollouts.jsonl` contains one checked rollout for each of the five example rows. Representative task checks: - `pt`: `<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>` — `format_ok=true`, reward `1.0`, visible-frame mask membership `true`. - `box`: `<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>` — `format_ok=true`, reward `1.0`, frame IoU `1.0`. - `ref`: `<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` — `format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT object and mask membership `true`. The artifact also contains visible/invisible `pt` coverage and an all-absent `ref` negative case, preserving the verifier's current unmatched-ID behavior. ## Compatibility Adds a new verifier/configuration and does not change unrelated environments. Example media is embedded as a small inline JPEG, so no employee or cluster-local paths are required. ## Stack Gym PR 1 of 2. Consolidates and supersedes the implementation scope of NVIDIA-NeMo#3504 after review history is preserved. ## Consolidated stack links ### Gym 1. [Gym NVIDIA-NeMo#3999 — SA-V tracking verifier](NVIDIA-NeMo#3999) 2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout identity](NVIDIA-NeMo#4000) ### NeMo-RL 1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model loading](NVIDIA-NeMo/RL#4472) 2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame contract](NVIDIA-NeMo/RL#4473) 3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing runtime](NVIDIA-NeMo/RL#4474) 4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI fixtures](NVIDIA-NeMo/RL#4475) 5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency pins](NVIDIA-NeMo/RL#4476) ### Child dependency refs - [MCore integration ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004), adapting [MCore #7278](NVIDIA/Megatron-LM#7278) and [MCore #7279](NVIDIA/Megatron-LM#7279) to the pinned base. - [MBridge integration ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004). --------- Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com> Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
|
Following up on my report in #7338 (same cluster): a second production caller that produces the mixed-dtype list, this time from RL rather than SFT. Setup: verl GRPO ( What the caller sees: on every rank and every step the list handed to Result: with a caller-side per-dtype split installed at runtime (bucket by dtype, sum of squared per-bucket norms, then the existing single all-reduce; per-bucket in-place scaling — the same contract as this PR, re-implemented as a monkeypatch because we are pinned to |
…oval-plan entries patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main) does the same index rebuild; #5835 merged it into dev only, so it is not in the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open; issue #7338) splits by dtype inside get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and at the call sites; README sections with the landed check and what to remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
…oval-plan entries patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main) does the same index rebuild; #5835 merged it into dev only, so it is not in the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open; issue #7338) splits by dtype inside get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and at the call sites; README sections with the landed check and what to remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
…oval-plan entries patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main) does the same index rebuild; #5835 merged it into dev only, so it is not in the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open; issue #7338) splits by dtype inside get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and at the call sites; README sections with the landed check and what to remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
…oval-plan entries patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main) does the same index rebuild; #5835 merged it into dev only, so it is not in the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open; issue #7338) splits by dtype inside get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and at the call sites; README sections with the landed check and what to remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
|
@Connor-XY, could you help get an optimizer-owner review for this PR and confirm what remains to get it through CI and the final review gates? The recent GRPO report provides another production example of mixed BF16/FP32 gradients. Its successful runs used an independently implemented dtype split on an older checkout, so they don’t validate this PR’s code. The last native two-H100 validation covered |
…a Megatron-LM branch instead of verl-side patches Per review: verl should not carry Megatron-Core patches. The runtime equivalents of NVIDIA/Megatron-LM#7279 and verl-project#8245 (verl/models/mcore/patch.py, their hook in the Megatron engine and the CPU tests) are removed; the script header and the doc point to linmuchuiyang/Megatron-LM@dsv4-pro-optimizer-fixes, which is the commit Megatron-Bridge 0.6.2 pins (43e45e13) plus those two fixes and the other pending Megatron-LM fixes the example needs (verl-project#4046, verl-project#7785, the native-fp32-param detach for optimizer_cpu_offload, the FlashAttention-4 import guard of verl-project#6545), one commit each. The script warns at start when the installed Megatron-LM lacks the per-dtype grad-norm kernels. Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
…d GB200 profiles (#8383) ### What does this PR do? Adds a DeepSeek-V4-**Pro** GRPO example and a "Running DeepSeek-V4-Pro at scale" section to the DeepSeek V4 guide. The Flash example and docs cover the 40-layer model on 8 nodes; Pro (61 layers, 384 routed experts, ~1.6 T parameters) runs through the same integration but needs a different scale and a few memory levers, and the first run otherwise costs a lot of node-hours of trial and error (it did for us). - `examples/grpo_trainer/run_deepseek_v4_pro_megatron.sh`: derived from `run_deepseek_v4_flash_megatron.sh` with identical algorithm / data / reward settings and two hardware profiles selected by an env toggle (`HW=h100`: 64 × 8 H100 80 GB, PP32 × EP8, rollout TP8 × DP8 × EP64; `HW=gb200`: 64 × 4 GB200, PP16 × EP16, rollout DP32 × EP32). Every knob that differs from Flash carries the reason in a comment. - `docs/advance/deepseek_v4_integration.rst`: topology (cross-node rollout replicas, PP layout with the hash layers in stage 0), the host-memory model under all-CPU optimizer offload and the three levers (bf16 grad buffers, precision-aware optimizer, `optimizer_offload_fraction`), checkpoint options and their sizes, rollout kernel choices for a native DSV4 checkpoint, what the DAPO reward measures on Pro, and the measured numbers for both profiles. - Megatron-LM: the precision-aware optimizer on a quantized checkpoint needs two Megatron-LM fixes that are still open — no-grad fp32 main-param shards (NVIDIA/Megatron-LM#8245; otherwise `optimizer_cpu_offload=True` raises `can't optimize a non-leaf Tensor`) and per-dtype grad-norm / clipping kernels (NVIDIA/Megatron-LM#7279; otherwise the mixed bf16/fp32 gradient list hits an illegal memory access). Per review they are not carried as verl-side patches: the script header and the doc point to `linmuchuiyang/Megatron-LM@dsv4-pro-optimizer-fixes`, which is the commit Megatron-Bridge 0.6.2 pins (43e45e13) plus those two fixes and the other pending Megatron-LM fixes the example needs (#4046 and #7785 for resuming the offloaded HybridDeviceOptimizer, the native-fp32-param detach for `optimizer_cpu_offload`, the FlashAttention-4 import guard of #6545), one commit each. The script warns at start when the installed Megatron-LM lacks the per-dtype grad-norm kernels. Dependencies called out in the script header and the doc: #8234 (cross-node vLLM data-parallel replicas) and #8235 (resuming an offloaded optimizer) — **both merged on 2026-10-26** (3dd97a0, 0bd668c), so verl `main` at or after those commits is all the example needs. ### Checklist Before Starting - [x] Search for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+deepseek+v4+pro — none; #6864 and the Flash example cover Flash only. - [x] Format the PR title as `[{modules}] {type}: {description}`. ### Test - `python3 tests/special_sanity/check_example_naming.py` and `check_docs_time_info.py` pass; `bash -n` on the script; both profiles expand to the intended topology. The Megatron-LM branch's own tests pass on 2 GPUs (`tests/unit_tests/optimizer/test_clip_grads.py` incl. #7279's mixed-dtype cases, `test_distrib_optimizer_grouped_quantized.py`). - GB200, Megatron-LM `linmuchuiyang/Megatron-LM@dsv4-pro-optimizer-fixes` bind-mounted over the image's tree, Megatron-Bridge 0.6.2, vLLM 0.29.0: - 4-layer truncated Pro, 8 nodes, GRPO 2 steps with the precision-aware optimizer (bf16 main grads, fp32 moments, all-CPU offload): with this PR's verl (da6444a, no verl-side Megatron-Core patches) 2/2 steps, rollout/actor log-prob Pearson 0.9998 / 0.9998, actor GPU peak 66.3 GB (fp32 grad buffers); with the image's pinned verl and bf16 grad buffers 2/2 steps, Pearson 0.9993 / 0.9992, 48.1 GB — in both settings identical to the MiB to the same run with the runtime patches instead of the branch. - 61-layer Pro, `HW=gb200` profile (64 nodes, PP16 × EP16, rollout DP32 × EP32, response 8192, R3, Marlin), 3 steps, with the runtime patches instead of the branch (verl `main@a4b9354` + #8234, same Megatron-LM content): steady step 960 / 972 s (first 1991 s), `gen` ≈500 s, `update_weights` ≈251 s, Pearson 0.9981 / 0.9983 / 0.9980, actor GPU 63.8 GB, head-node host peak 929 GB. - H100 (64 × 8 H100 80 GB), `HW=h100` profile with the same Megatron-LM branch: 3/3 steps in 1 h 30 min with the runtime patches disabled (`VERL_MEGATRON_OPTIMIZER_COMPAT=0`) and the branch's `megatron/core` bind-mounted over the venv: steady step 700 / 672 s (first 2612 s with JIT), `gen` 284 / 274 s, `update_weights` 238 s, Pearson 0.9983 / 0.9984 / 0.9982, actor GPU 26.4 GB, host peak 560 / 553 GB on the two stage-0 nodes (236–406 GB elsewhere) — unchanged from the run with the patches. - Earlier validation on verl `main@a4b9354` + #8234 with the runtime patches (both profiles, 3 steps each, including the deep_gemm-vs-Marlin findings behind the script's Marlin default, see #8381) and on pinned verl `1d367a4f` (GB200) / `45a40312` (H100), including a 40-step segmented run, save/resume at 64 and 128 nodes and the reward analysis quoted in the doc, is summarised in the doc section. ### API and Usage Example No API change. ```bash # 64 nodes x 8 H100 HW=h100 MODEL_PATH=... TRAIN_FILE=... TEST_FILE=... bash examples/grpo_trainer/run_deepseek_v4_pro_megatron.sh # 64 nodes x 4 GB200, actor from a Megatron dist-checkpoint, precision-aware optimizer so that 64 nodes can save HW=gb200 DIST_CKPT_PATH=... PRECISION_AWARE_OPTIMIZER=True SAVE_FREQ=20 bash examples/grpo_trainer/run_deepseek_v4_pro_megatron.sh ``` ### Design & Code Changes - One script, two profiles: the naming check forbids hardware variants in the file name, so the profile is an env toggle that only sets defaults; every value can still be overridden. - No Megatron-Core patches in verl: the Megatron-LM fixes live on the branch above and drop out once the upstream PRs merge (the branch is a plain superset of Megatron-Bridge 0.6.2's pinned commit). ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [x] Add unit or end-to-end test(s): not feasible in CI (64-node, 1.6 T model); validated by the runs listed under Test. - [ ] Once your PR is ready for CI, send a message in the `ci-request` channel. --------- Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com> Co-authored-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
What does this PR do?
Group gradients by dtype before fused L2-norm and clipping-scale calls, so mixed BF16/FP32 lists produce the intended norm and in-place scaling.
The fused backend tested here interprets a list using its first tensor's dtype. For example,
[BF16[128], FP32[128]]filled with1/64returns0.2165063471instead of0.25. Reversing the order can read beyond the logical BF16 view. Scaling has an independent defect: even with coefficient0.5supplied directly, mixed lists leave some elements unscaled or modify padding beyond the gradient view. The reproductions use padded backing allocations.The change:
.gradis also exercised by the mixed-dtype determinism tests.The existing single-device gradient-list contract is retained. No kernel implementation or optimizer-state layout changes are required.
Validation
The GPU component results below describe the original fix at
333ecc9. Validation of the review cleanup is recorded separately below.2.11.0+cu130and the Transformer Engine backend: all 48 norm boundary cases per rank matched first-dtype byte interpretation, with 15 incorrect logical norms. Both scaling APIs failed on both mixed orders. Bucketed controls preserved padding and matched the logical reference.tools/autoformat.sh: Black, isort, pylint, and Ruff passed. Mypy could not resolve Torch/TE/Apex/pytest imports in the isolated local lint environment.python3 tools/check_kernel_determinism_coverage.py --base-ref origin/main: passed.Review cleanup (
09f6e03): DFW job19072876completed on two H100 80GB GPUs with PyTorch2.11.0+cu130and Transformer Engine2.15.0+42b84005. All 17 unit cases and both mixed-dtype determinism cases passed on each rank, with no failures, errors, or skips. Each determinism case includes four bit-exact replays with side-stream contention. The runner verified exact source hashes and native TE dispatch; unrelated top-level suite fixtures were excluded. Local formatting/lint and the determinism coverage check also pass.The reporter in #7338 also completed a three-step H100 training smoke after adapting the caller patch to their older
devcheckout. This supports the correction in that configuration, rather than validating a complete checkout of this PR's current head.This establishes the component defect and correction; it does not claim a measured full-training quality impact or that the fused API promises heterogeneous-list support.
Issue tracking
Related issue: #7338.
Contribution process
Pre-checks
tools/autoformat.shon my PRNo training recipe or public API changes; no functional-test goldens or user documentation updates are included.