Skip to content

Fix mixed-dtype gradient norms and clipping - #7279

Open
aroshanghias-nvd wants to merge 3 commits into
NVIDIA:mainfrom
aroshanghias-nvd:fix/mixed-dtype-gradient-norm
Open

aroshanghias-nvd wants to merge 3 commits into
NVIDIA:mainfrom
aroshanghias-nvd:fix/mixed-dtype-gradient-norm

Conversation

@aroshanghias-nvd

@aroshanghias-nvd aroshanghias-nvd commented Sep 12, 2026 •

Copy link
Copy Markdown
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Group gradients by dtype before fused L2-norm and clipping-scale calls, so mixed BF16/FP32 lists produce the intended norm and in-place scaling.

The fused backend tested here interprets a list using its first tensor's dtype. For example, [BF16[128], FP32[128]] filled with 1/64 returns 0.2165063471 instead of 0.25. Reversing the order can read beyond the logical BF16 view. Scaling has an independent defect: even with coefficient 0.5 supplied directly, mixed lists leave some elements unscaled or modify padding beyond the gradient view. The reproductions use padded backing allocations.

The change:

  • Buckets L2-norm inputs locally and sums their squared norms before the existing distributed reductions. This preserves collective order for empty ranks and ranks with different dtype sets, and accumulates into the existing zero-valued CUDA tensor.
  • Buckets both scalar- and tensor-coefficient scaling calls, and avoids calling fused kernels with empty lists.
  • Adds focused numerical regressions for both mixed orders, decoupled gradients, scalar- and tensor-coefficient clipping, a non-vector-aligned length, backing-storage preservation, empty gradients, and asymmetric rank ownership. Ordinary .grad is also exercised by the mixed-dtype determinism tests.
  • Adds mixed-dtype norm/clipping bit-exact replay under side-stream contention to the existing registered optimizer determinism tests.

The existing single-device gradient-list contract is retained. No kernel implementation or optimizer-state layout changes are required.

Validation

The GPU component results below describe the original fix at 333ecc9. Validation of the review cleanup is recorded separately below.

  • Native two-rank GB200 investigation with PyTorch 2.11.0+cu130 and the Transformer Engine backend: all 48 norm boundary cases per rank matched first-dtype byte interpretation, with 15 incorrect logical norms. Both scaling APIs failed on both mixed orders. Bucketed controls preserved padding and matched the logical reference.
  • Matched caller regression: the original had 24 numerical failures in 96 cases per rank; the corrected caller passed those cases plus four empty-list cases per rank. Mixed-buffer and FP32-buffer DDP/DistributedOptimizer/FusedAdam fixtures each completed eight arms per rank.
  • The production file on the PR base is byte-identical to the original file used for that investigation. The PR adds typing and repository formatting to the validated implementation.
  • Repository tools/autoformat.sh: Black, isort, pylint, and Ruff passed. Mypy could not resolve Torch/TE/Apex/pytest imports in the isolated local lint environment.
  • python3 tools/check_kernel_determinism_coverage.py --base-ref origin/main: passed.
  • Exact PR tests on two GB200 ranks with the native TE backend: the original caller produced 29 expected failures out of 105 unit cases per rank; the fixed caller passed all 105 with no skips or errors. Both mixed-dtype determinism cases passed on both ranks, including four bit-exact replays with side-stream contention. Test dependencies were staged separately from the pinned runtime; unrelated top-level suite fixtures were excluded for this focused run.

Review cleanup (09f6e03): DFW job 19072876 completed on two H100 80GB GPUs with PyTorch 2.11.0+cu130 and Transformer Engine 2.15.0+42b84005. All 17 unit cases and both mixed-dtype determinism cases passed on each rank, with no failures, errors, or skips. Each determinism case includes four bit-exact replays with side-stream contention. The runner verified exact source hashes and native TE dispatch; unrelated top-level suite fixtures were excluded. Local formatting/lint and the determinism coverage check also pass.

The reporter in #7338 also completed a three-step H100 training smoke after adapting the caller patch to their older dev checkout. This supports the correction in that configuration, rather than validating a complete checkout of this PR's current head.

This establishes the component defect and correction; it does not claim a measured full-training quality impact or that the fused API promises heterogeneous-list support.

Issue tracking

Related issue: #7338.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added bit-exact determinism tests under the existing registry entry
  • I have added proper typing to my code
  • I have added relevant documentation
  • I have run tools/autoformat.sh on my PR

No training recipe or public API changes; no functional-test goldens or user documentation updates are included.

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Connor-XY

Copy link
Copy Markdown
Contributor

Linking the community report in #7338: it describes the same mixed BF16/FP32 norm/clipping caller issue and now includes a single-GPU reproducer plus an H100 training case. This PR appears to cover the proposed caller-side correction, including both clipping-scale paths.

@aroshanghias-nvd, please link that report in the PR description and let us know what remains before this is ready for review. The reporter's configuration would provide useful validation alongside the component tests already described here.

Copy link
Copy Markdown
Author

Author review is complete, and I’ve linked #7338. The component and determinism tests pass as described. Validation of commit 333ecc9 in the reporter’s H100 training configuration remains outstanding and would complement those tests. I’m marking this ready for review.

@Connor-XY

Copy link
Copy Markdown
Contributor

@wujingyue, could you review the mixed-dtype norm and clipping changes now that this is ready? The reporter has added a successful three-step H100 training smoke in #7338 (comment). They adapted the caller patch from 333ecc9 to their older dev checkout (43e45e1), so this supports the fix in their configuration rather than validating the complete PR checkout at its current head.

@wujingyue wujingyue left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread megatron/core/optimizer/clip_grads.py Outdated
l2_norm_impl, dummy_overflow_buf, [dtype_grads], False
)
# Combine local squared norms before the existing collectives.
if index == 0:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simply initialize total_norm to 0.0 and get rid of this condition?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified to accumulate into the existing zero-valued CUDA tensor and removed the first-group conditional. Keeping the tensor initialization, rather than a Python float, also handles ranks with no local gradients when they reach the existing reductions. The reductions remain outside the grouping loop.

Comment thread megatron/core/optimizer/clip_grads.py Resolved
Comment thread megatron/core/optimizer/clip_grads.py Outdated
total_norm = grad_norm**norm_type
for index, dtype_grads in enumerate(_group_grads_by_dtype(grads_for_norm)):
grad_norm, _ = multi_tensor_applier(
l2_norm_impl, dummy_overflow_buf, [dtype_grads], False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's hard for readers to figure out what False mean. Consider a kwarg.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added # no per-tensor norms to explain the flag. I kept it positional because the supported local multi_tensor_applier wrapper accepts *args, not keyword arguments (megatron/core/utils.py).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multi_tensor_applier wrapper accepts *args, not keyword arguments (megatron/core/utils.py).

Got it -- that's a bummer but worth a separate PR :)

Comment thread megatron/core/optimizer/clip_grads.py Resolved
Comment on lines +422 to +423
notes="multi_tensor l2norm / scale (TE, apex or local fallback), including mixed-dtype "
"caller bucketing and padded-storage replay, and fused Adam; "

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wording was intended to document the added mixed-dtype replay coverage. The existing optimizer entry already points to that test file, so I removed the redundant manifest change. The coverage checker still passes with the new replay tests.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test looks pretty verbose. For coverage, I'd just test one config that triggers mixed grad_dtype.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reduced the new numerical test matrix from 102 cases to 14, using one gradient length and the precision-aware decoupled-gradient path. I retained both dtype orders because they exercise different byte-width failures, and both coefficient APIs because norm correctness alone does not guarantee correct scaling. The small norm layout matrix retains empty and disjoint rank ownership coverage. Ordinary .grad remains exercised by the mixed-dtype determinism tests.


@pytest.mark.parametrize('use_decoupled_grad', [False, True])
@pytest.mark.parametrize('tensor_coefficient', [False, True])
def test_empty_gradients_do_not_launch_kernel(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is arguably testing implementation not behavior. It makes sense to verify grad clipping doesn't crash with empty gradients, but whether a kernel gets launched behind the scene is an implementation detail that I'd rather not test.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Replaced the kernel-launch assertion with a direct empty-gradient clipping call and CUDA synchronization, so the test checks successful completion rather than how the operation is implemented.

@wujingyue

Copy link
Copy Markdown
Contributor

Again, this PR should go to @NVIDIA/mcore-optimizer. I happened to touch the optimizer code recently for FSDP, but I don't own the code.

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
assert num_zeros == grad.numel() - 1


_DTYPES = [(torch.bfloat16, torch.float32), (torch.float32, torch.bfloat16)]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hard to read. The name doesn't say why the dtypes are in tuples.

Also, it's on the boundary of https://testing.googleblog.com/2019/12/testing-on-toilet-tests-too-dry-make.html. Consider letting each test manages its own parametrization with a well-documented intention.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inlined the dtype tuples in each test and documented the intention: fused kernels dispatch using the first tensor's dtype, so both mixed orders exercise distinct failure modes.

grads, grad_stats_parallel_group=torch.distributed.group.WORLD
)

assert float(actual) == pytest.approx(float(squared_norm.sqrt()), rel=2e-6, abs=1e-8)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

torch.testing.assert_close instead of reinventing another way to compare?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the norm comparison with torch.testing.assert_close, retaining the existing relative and absolute tolerances.


@pytest.mark.parametrize('tensor_coefficient', [False, True])
def test_clip_empty_gradients(self, tensor_coefficient):
if tensor_coefficient and clip_grads.multi_tensor_scale_tensor_impl is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if tensor_coefficient and clip_grads.multi_tensor_scale_tensor_impl is None:

IIUC, CI installs TE so this check is not so relevant. It's like we could ask every test to skipif not torch.cuda.is_available but nobody really cares :)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the backend skip checks from the mixed-dtype tests, including both tensor-coefficient checks. CI supplies Transformer Engine, so missing backend support should surface as a test failure.

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com>
@aroshanghias-nvd

Copy link
Copy Markdown
Author

Addressed the remaining test feedback: each test now documents its dtype-order parameters, the norm assertion uses torch.testing.assert_close, and the backend skip checks are removed. The cleanup changes only tests; production code and determinism tests are unchanged. Local formatting/lint, determinism coverage and the 14-case CPU reference check pass. The existing two-H100 native validation remains the evidence for 09f6e03; no fresh native run was performed for this test cleanup.

@wujingyue, could you refresh your approval after checking this cleanup? @NVIDIA/mcore-optimizer, could an optimizer owner review the caller fix so this can advance through the owner and final-review gates?

@wujingyue

Copy link
Copy Markdown
Contributor

Assigned the PR to @Connor-XY, who was the on-call when the PR was initially sent out.

@wujingyue

Copy link
Copy Markdown
Contributor

@wujingyue, could you refresh your approval after checking this cleanup? @NVIDIA/mcore-optimizer, could an optimizer owner review the caller fix so this can advance through the owner and final-review gates?

No need to -- approval sticks. I assigned this to @Connor-XY who'll help you get other approvals.

khazic pushed a commit to khazic/Gym-lao that referenced this pull request Oct 15, 2026
## Summary
- Add the SA-V tracking verifier and its server configuration.
- Declare the SA-V environment servers used by video RL.
- Add portable five-row example data, schema, metrics, and checked
rollout artifacts.

## Why
This is the Gym-side verifier foundation for mixed SA-V tracking and
CapRL video training in NeMo-RL. No separate issue is needed; this is
part of the Super 3.5 video RL upstreaming stack.

## Validation
- Runtime evidence: the [healthy 16-node async `r18` W&B
run](https://wandb.ai/adlr/Nemotron-omni-RL-debug/runs/oci-hsg-cs-001_n3post_main_1d76d243b_vllm029_super35_video_step120_async16n_trainmb45056_expandable_uncapped_r18_20261003)
shows repeated GRPO cycles, stable TMPE, and increasing reward on the
equivalent video implementation.
- `gym dataset collate ... +mode=example_validation`: 5 examples;
generated `data/example_metrics.json`.
- Focused validation: 25 verifier tests and all 5 example rows against
`task_data.py` passed.
- Ruff format and lint passed for the changed Python files.
- Both Gym stack ranges pass `git diff --check`.

## Rollouts
`data/example_rollouts.jsonl` contains one checked rollout for each of
the five example rows. Representative task checks:

- `pt`:
`<answer>{"tracks":[{"point":1,"frame":1,"xy":[500,500],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, visible-frame mask membership `true`.
- `box`:
`<answer>{"tracks":[{"box":1,"frame":1,"bbox":[250,250,750,750],"visible":true}]}</answer>`
— `format_ok=true`, reward `1.0`, frame IoU `1.0`.
- `ref`:
`<answer>{"tracks":[{"id":7,"frame":1,"xy":[500,500]}]}</answer>` —
`format_ok=true`, reward `1.0`, predicted id `7` assigned to the GT
object and mask membership `true`.

The artifact also contains visible/invisible `pt` coverage and an
all-absent `ref` negative case, preserving the verifier's current
unmatched-ID behavior.

## Compatibility
Adds a new verifier/configuration and does not change unrelated
environments. Example media is embedded as a small inline JPEG, so no
employee or cluster-local paths are required.

## Stack
Gym PR 1 of 2. Consolidates and supersedes the implementation scope of
NVIDIA-NeMo#3504 after review history is preserved.

## Consolidated stack links

### Gym
1. [Gym NVIDIA-NeMo#3999 — SA-V tracking
verifier](NVIDIA-NeMo#3999)
2. [Gym NVIDIA-NeMo#4000 — R3 routing replay and rollout
identity](NVIDIA-NeMo#4000)

### NeMo-RL
1. [NeMo-RL NVIDIA-NeMo#4472 — Super video data and model
loading](NVIDIA-NeMo/RL#4472)
2. [NeMo-RL NVIDIA-NeMo#4473 — RL/MOPD video frame
contract](NVIDIA-NeMo/RL#4473)
3. [NeMo-RL NVIDIA-NeMo#4474 — sync/async Gym routing
runtime](NVIDIA-NeMo/RL#4474)
4. [NeMo-RL NVIDIA-NeMo#4475 — SA-V pin and CI
fixtures](NVIDIA-NeMo/RL#4475)
5. [NeMo-RL NVIDIA-NeMo#4476 — final dependency
pins](NVIDIA-NeMo/RL#4476)

### Child dependency refs
- [MCore integration
ref](https://github.com/ehosseiniasl/Megatron-LM/tree/ehsan/video-r3-mtp-gradient-integration-20261004),
adapting [MCore #7278](NVIDIA/Megatron-LM#7278)
and [MCore #7279](NVIDIA/Megatron-LM#7279) to
the pinned base.
- [MBridge integration
ref](https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/ehsan/video-r3-mcore-pin-integration-20261004).

---------

Signed-off-by: ehosseiniasl <ehsan.hosseiniasl@gmail.com>
Co-authored-by: Pulkit Kumar <pulkitk@nvidia.com>
@linmuchuiyang

Copy link
Copy Markdown

Following up on my report in #7338 (same cluster): a second production caller that produces the mixed-dtype list, this time from RL rather than SFT.

Setup: verl GRPO (main@45a40312) on the Megatron-Bridge backend, Megatron-LM 43e45e13 (Megatron-Bridge 0.6.2's .dev.commit, which predates this PR), DeepSeek-V4-Pro (61 layers, 1.57 T) on 64 H100-80GB nodes, actor PP32/EP8/DP2, use_precision_aware_optimizer=True, main_grads_dtype=bf16, DDP grad_reduce_in_fp32=False, optimizer_cpu_offload=True (HybridDeviceOptimizer).

What the caller sees: on every rank and every step the list handed to get_grad_norm_fp32 / clip_grad_by_total_norm_fp32 is mixed. DSv4's native fp32 parameters (hc alpha/bias, ape, attn_sink) keep fp32 grads while everything else has bf16 decoupled grads. Logged per rank by our shim:

groups={'torch.float32': 1, 'torch.bfloat16': 37}    # Pro, PP32 (37–66 bf16 tensors depending on the stage)
groups={'torch.float32': 9, 'torch.bfloat16': 321}   # Flash (43 layers), PP8

Result: with a caller-side per-dtype split installed at runtime (bucket by dtype, sum of squared per-bucket norms, then the existing single all-reduce; per-bucket in-place scaling — the same contract as this PR, re-implemented as a monkeypatch because we are pinned to 43e45e13, so this is not a validation of this PR's code) the runs are healthy: 5-, 3-, 2-, 3- and 13-step GRPO runs, rollout/actor log-prob Pearson 0.994–0.998, HF checkpoint save/resume continuing the metrics. We did not run the RL path without the split; the crash for this parameter mix is the one documented in #7338.

avigyabb added a commit to avigyabb/SkyRL that referenced this pull request Oct 27, 2026
…oval-plan entries

patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main)
does the same index rebuild; #5835 merged it into dev only, so it is not in
the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open;
issue #7338) splits by dtype inside get_grad_norm_fp32 /
clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and
at the call sites; README sections with the landed check and what to remove.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
avigyabb added a commit to avigyabb/SkyRL that referenced this pull request Oct 27, 2026
…oval-plan entries

patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main)
does the same index rebuild; #5835 merged it into dev only, so it is not in
the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open;
issue #7338) splits by dtype inside get_grad_norm_fp32 /
clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and
at the call sites; README sections with the landed check and what to remove.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
avigyabb added a commit to avigyabb/SkyRL that referenced this pull request Oct 27, 2026
…oval-plan entries

patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main)
does the same index rebuild; #5835 merged it into dev only, so it is not in
the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open;
issue #7338) splits by dtype inside get_grad_norm_fp32 /
clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and
at the call sites; README sections with the landed check and what to remove.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC
avigyabb added a commit to avigyabb/SkyRL that referenced this pull request Oct 27, 2026
…oval-plan entries

patch_distopt_mixed_dtype_group_index: NVIDIA/Megatron-LM#2872 (open, main)
does the same index rebuild; #5835 merged it into dev only, so it is not in
the main pin. patch_clip_grads_mixed_dtype: NVIDIA/Megatron-LM#7279 (open;
issue #7338) splits by dtype inside get_grad_norm_fp32 /
clip_grad_by_total_norm_fp32 with the same math. TODOs in both modules and
at the call sites; README sections with the landed check and what to remove.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F9VEAjazBNx4WcuvyfY3BC

Copy link
Copy Markdown
Author

@Connor-XY, could you help get an optimizer-owner review for this PR and confirm what remains to get it through CI and the final review gates?

The recent GRPO report provides another production example of mixed BF16/FP32 gradients. Its successful runs used an independently implemented dtype split on an older checkout, so they don’t validate this PR’s code.

The last native two-H100 validation covered 09f6e03; the current head has subsequent test-only cleanup. Would a fresh native run on the current head be useful, or should we rely on the project CI once it runs?

linmuchuiyang added a commit to linmuchuiyang/verl that referenced this pull request Nov 5, 2026
…a Megatron-LM branch instead of verl-side patches

Per review: verl should not carry Megatron-Core patches. The runtime
equivalents of NVIDIA/Megatron-LM#7279 and verl-project#8245 (verl/models/mcore/patch.py,
their hook in the Megatron engine and the CPU tests) are removed; the script
header and the doc point to linmuchuiyang/Megatron-LM@dsv4-pro-optimizer-fixes,
which is the commit Megatron-Bridge 0.6.2 pins (43e45e13) plus those two fixes
and the other pending Megatron-LM fixes the example needs (verl-project#4046, verl-project#7785, the
native-fp32-param detach for optimizer_cpu_offload, the FlashAttention-4 import
guard of verl-project#6545), one commit each. The script warns at start when the installed
Megatron-LM lacks the per-dtype grad-norm kernels.

Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
wuxibin89 pushed a commit to verl-project/verl that referenced this pull request Nov 5, 2026
…d GB200 profiles (#8383)

### What does this PR do?

Adds a DeepSeek-V4-**Pro** GRPO example and a "Running DeepSeek-V4-Pro
at scale" section to the DeepSeek V4 guide. The Flash example and docs
cover the 40-layer model on 8 nodes; Pro (61 layers, 384 routed experts,
~1.6 T parameters) runs through the same integration but needs a
different scale and a few memory levers, and the first run otherwise
costs a lot of node-hours of trial and error (it did for us).

- `examples/grpo_trainer/run_deepseek_v4_pro_megatron.sh`: derived from
`run_deepseek_v4_flash_megatron.sh` with identical algorithm / data /
reward settings and two hardware profiles selected by an env toggle
(`HW=h100`: 64 × 8 H100 80 GB, PP32 × EP8, rollout TP8 × DP8 × EP64;
`HW=gb200`: 64 × 4 GB200, PP16 × EP16, rollout DP32 × EP32). Every knob
that differs from Flash carries the reason in a comment.
- `docs/advance/deepseek_v4_integration.rst`: topology (cross-node
rollout replicas, PP layout with the hash layers in stage 0), the
host-memory model under all-CPU optimizer offload and the three levers
(bf16 grad buffers, precision-aware optimizer,
`optimizer_offload_fraction`), checkpoint options and their sizes,
rollout kernel choices for a native DSV4 checkpoint, what the DAPO
reward measures on Pro, and the measured numbers for both profiles.

- Megatron-LM: the precision-aware optimizer on a quantized checkpoint
needs two Megatron-LM fixes that are still open — no-grad fp32
main-param shards (NVIDIA/Megatron-LM#8245; otherwise
`optimizer_cpu_offload=True` raises `can't optimize a non-leaf Tensor`)
and per-dtype grad-norm / clipping kernels (NVIDIA/Megatron-LM#7279;
otherwise the mixed bf16/fp32 gradient list hits an illegal memory
access). Per review they are not carried as verl-side patches: the
script header and the doc point to
`linmuchuiyang/Megatron-LM@dsv4-pro-optimizer-fixes`, which is the
commit Megatron-Bridge 0.6.2 pins (43e45e13) plus those two fixes and
the other pending Megatron-LM fixes the example needs (#4046 and #7785
for resuming the offloaded HybridDeviceOptimizer, the native-fp32-param
detach for `optimizer_cpu_offload`, the FlashAttention-4 import guard of
#6545), one commit each. The script warns at start when the installed
Megatron-LM lacks the per-dtype grad-norm kernels.

Dependencies called out in the script header and the doc: #8234
(cross-node vLLM data-parallel replicas) and #8235 (resuming an
offloaded optimizer) — **both merged on 2026-10-26** (3dd97a0,
0bd668c), so verl `main` at or after those commits is all the example
needs.

### Checklist Before Starting

- [x] Search for similar PRs:
https://github.com/verl-project/verl/pulls?q=is%3Apr+deepseek+v4+pro —
none; #6864 and the Flash example cover Flash only.
- [x] Format the PR title as `[{modules}] {type}: {description}`.

### Test

- `python3 tests/special_sanity/check_example_naming.py` and
`check_docs_time_info.py` pass; `bash -n` on the script; both profiles
expand to the intended topology. The Megatron-LM branch's own tests pass
on 2 GPUs (`tests/unit_tests/optimizer/test_clip_grads.py` incl. #7279's
mixed-dtype cases, `test_distrib_optimizer_grouped_quantized.py`).
- GB200, Megatron-LM
`linmuchuiyang/Megatron-LM@dsv4-pro-optimizer-fixes` bind-mounted over
the image's tree, Megatron-Bridge 0.6.2, vLLM 0.29.0:
- 4-layer truncated Pro, 8 nodes, GRPO 2 steps with the precision-aware
optimizer (bf16 main grads, fp32 moments, all-CPU offload): with this
PR's verl (da6444a, no verl-side Megatron-Core patches) 2/2 steps,
rollout/actor log-prob Pearson 0.9998 / 0.9998, actor GPU peak 66.3 GB
(fp32 grad buffers); with the image's pinned verl and bf16 grad buffers
2/2 steps, Pearson 0.9993 / 0.9992, 48.1 GB — in both settings identical
to the MiB to the same run with the runtime patches instead of the
branch.
- 61-layer Pro, `HW=gb200` profile (64 nodes, PP16 × EP16, rollout DP32
× EP32, response 8192, R3, Marlin), 3 steps, with the runtime patches
instead of the branch (verl `main@a4b9354` + #8234, same Megatron-LM
content): steady step 960 / 972 s (first 1991 s), `gen` ≈500 s,
`update_weights` ≈251 s, Pearson 0.9981 / 0.9983 / 0.9980, actor GPU
63.8 GB, head-node host peak 929 GB.
- H100 (64 × 8 H100 80 GB), `HW=h100` profile with the same Megatron-LM
branch: 3/3 steps in 1 h 30 min with the runtime patches disabled
(`VERL_MEGATRON_OPTIMIZER_COMPAT=0`) and the branch's `megatron/core`
bind-mounted over the venv: steady step 700 / 672 s (first 2612 s with
JIT), `gen` 284 / 274 s, `update_weights` 238 s, Pearson 0.9983 / 0.9984
/ 0.9982, actor GPU 26.4 GB, host peak 560 / 553 GB on the two stage-0
nodes (236–406 GB elsewhere) — unchanged from the run with the patches.
- Earlier validation on verl `main@a4b9354` + #8234 with the runtime
patches (both profiles, 3 steps each, including the deep_gemm-vs-Marlin
findings behind the script's Marlin default, see #8381) and on pinned
verl `1d367a4f` (GB200) / `45a40312` (H100), including a 40-step
segmented run, save/resume at 64 and 128 nodes and the reward analysis
quoted in the doc, is summarised in the doc section.

### API and Usage Example

No API change.

```bash
# 64 nodes x 8 H100
HW=h100 MODEL_PATH=... TRAIN_FILE=... TEST_FILE=... bash examples/grpo_trainer/run_deepseek_v4_pro_megatron.sh
# 64 nodes x 4 GB200, actor from a Megatron dist-checkpoint, precision-aware optimizer so that 64 nodes can save
HW=gb200 DIST_CKPT_PATH=... PRECISION_AWARE_OPTIMIZER=True SAVE_FREQ=20 bash examples/grpo_trainer/run_deepseek_v4_pro_megatron.sh
```

### Design & Code Changes

- One script, two profiles: the naming check forbids hardware variants
in the file name, so the profile is an env toggle that only sets
defaults; every value can still be overridden.
- No Megatron-Core patches in verl: the Megatron-LM fixes live on the
branch above and drop out once the upstream PRs merge (the branch is a
plain superset of Megatron-Bridge 0.6.2's pinned commit).

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [x] Add unit or end-to-end test(s): not feasible in CI (64-node, 1.6 T
model); validated by the runs listed under Test.
- [ ] Once your PR is ready for CI, send a message in the `ci-request`
channel.

---------

Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
Co-authored-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants