Skip to content

Commit a53e2eb

Browse files
committed
Merge upstream/main into feat/sc-resiliency-04-restart-readmission
PR3 (#3591) was SQUASH-merged into main as b3b6713, so none of its commits are ancestors of main while PR4 still carries all of them. Git therefore sees PR3's whole diff as independently added on both sides, which is why all 16 conflicts name b3b6713 and why the PR showed CONFLICTING despite the content being identical. That made the classification, not the content, the work. For each conflicted file: is main's version byte-identical to PR3's head (3d9ce21), and does PR4 add anything beyond it? Three groups fell out. GROUP A -- pure squash artefacts, resolved by taking OURS (10 files) fleet_health.py, collective_weight_synchronizer.py, membership.py, nccl_reshard_weight_synchronizer.py, grpo_sc_generation_shard_recovery.sh, test_watchdog_pump.py, test_membership.py, test_reconcile_communicator.py, test_reshard_rebuild.py, test_weight_synchronizer.py main == PR3 exactly and no other PR touched them, so PR4's side is main's content plus PR4's delta. Taking ours loses nothing. GROUP B -- PR4 contributes nothing, resolved by taking THEIRS (2 files) single_controller_utils/setup.py (#3480, #3727, #3821 on top of PR3) tests/unit/single_controller/test_refit_recovery.py (#3480 on top of PR3) GROUP C -- genuine merges (4 files), one per upstream PR below. The six upstream PRs that contributed real content, and what each needed: #3480 recover replay buffer from native TQ checkpoints single_controller.py: rollout_recovery imports. Kept alongside ours. setup.py, test_refit_recovery.py, L1 harness: group B / additive. #3765 log toolcall and thinktag violation rate single_controller.py: VIOLATION_TAG_KEYS. Auto-merged, verified present. #3727 support non-colocated MInf single_controller.py: MegatronGeneration import, kept alongside ours. L1 harness: grpo_megatron_generation_gym_single_controller.sh entry. #3821 warm-start the value model from a critic-pretrain checkpoint config.py: the max_num_epochs validator. Ours only adds restart_dead_shards to FleetHealthConfig, so both survive; verified the field landed in the right class and the validator is intact. #3655 nemo-lens telemetry vllm_generation.py: the @trace_fn decorator on generate. Ours adds restart_shard in a different region; both kept. #3839 pause generation during in-flight refit vllm_generation.py: pause_generation_for_refit / resume_generation_after_refit. Auto-merged, verified present -- worth knowing it exists, since it pauses engines around a refit and this PR restarts them. Verified after resolving: no conflict markers; all four lint hooks clean (the single pyrefly error is the pre-existing unrelated transfer_queue import); 1122 unit tests pass; both submodule pointers and uv.lock/pyproject byte-identical to main. Both sides' work was checked individually rather than assumed: EngineSupervisor wiring, restart_dead_shards, restart_shard, recreate_worker, desired_membership and the report_refit call on our side; the six items above on main's. Note for anyone reproducing locally: #3655 adds a nemo-lens dependency that the pre-merge container image does not carry, so tests fail at import with ModuleNotFoundError: nemo until the venv is refreshed. Plain upstream/main fails the same way in that image; it is not a merge defect. Signed-off-by: asolergibert <asolergibert@nvidia.com>
2 parents 255d13f + 347fcbc commit a53e2eb

187 files changed

Lines changed: 20717 additions & 3341 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/cicd-main.yml‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -284,11 +284,20 @@ jobs:
284284
fi
285285
echo "image_tag=$IMAGE_TAG" | tee -a "$GITHUB_OUTPUT"
286286
287+
ephemeral-runner-routing:
288+
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_ephemeral_runner_routing.yml@d0878a142525ff79366ca96fbc0b306cd8d64670 # v1.9.0
289+
with:
290+
lane: ${{ vars.NEMO_CI_EPHEMERAL_LANE }}
291+
legacy_label: ${{ vars.NON_NVIDIA_H100_RUNNER }}
292+
run_id: ${{ github.run_id }}
293+
run_attempt: ${{ github.run_attempt }}
294+
287295
org-member-pre-flight:
288-
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cicd_preflight.yml@cf5acebace78c7c339bc1f804357a991336f1c4d # v1.8.10
296+
needs: [ephemeral-runner-routing]
297+
uses: NVIDIA-NeMo/FW-CI-templates/.github/workflows/_cicd_preflight.yml@d0878a142525ff79366ca96fbc0b306cd8d64670 # v1.9.0
289298
with:
290299
default_runner_prefix: ${{ vars.DEFAULT_H100_RUNNER }}
291-
non_nvidia_runner_prefix: ${{ vars.NON_NVIDIA_H100_RUNNER }}
300+
non_nvidia_runner_prefix: ${{ needs.ephemeral-runner-routing.outputs.runner_label }}
292301
default_test_data_path: ${{ vars.DEFAULT_H100_TEST_DATA_PATH }}
293302
non_nvidia_test_data_path: ${{ vars.NON_NVIDIA_H100_TEST_DATA_PATH }}
294303
default_registry: ${{ vars.DEFAULT_H100_CONTAINER_REGISTRY }}

‎.gitmodules‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
[submodule "3rdparty/Automodel-workspace/Automodel"]
77
path = 3rdparty/Automodel-workspace/Automodel
88
url = https://github.com/NVIDIA-NeMo/Automodel.git
9-
branch = main
9+
branch = r0.6.0
1010
shallow = true
1111
[submodule "3rdparty/Gym-workspace/Gym"]
1212
path = 3rdparty/Gym-workspace/Gym
Submodule Automodel updated 2191 files

‎docker/Dockerfile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,12 @@ ENV CPLUS_INCLUDE_PATH=/usr/local/cuda/include/cccl
186186
# Verify with: python -c "import transformer_engine.pytorch as te; print(te.get_cudnn_version())"
187187
ENV CUDNN_HOME=/opt/nemo_rl_venv/lib/python3.13/site-packages/nvidia/cudnn
188188
ENV CUDNN_PATH=/opt/nemo_rl_venv/lib/python3.13/site-packages/nvidia/cudnn
189-
ENV LD_LIBRARY_PATH="/opt/nemo_rl_venv/lib/python3.13/site-packages/nvidia/cudnn/lib:${LD_LIBRARY_PATH}"
189+
# TileLang's $ORIGIN-relative RUNPATH breaks when uv installs worker venvs in
190+
# symlink mode: libtvm_compiler.so resolves into TileLang's cache directory and
191+
# cannot find libz3.so in Z3's separate cache directory.
192+
# TODO: Remove this workaround once the upstream issue is resolved.
193+
# See https://github.com/tile-ai/tilelang/issues/3075
194+
ENV LD_LIBRARY_PATH="/opt/nemo_rl_venv/lib/python3.13/site-packages/z3/lib:/opt/nemo_rl_venv/lib/python3.13/site-packages/nvidia/cudnn/lib:${LD_LIBRARY_PATH}"
190195

191196
# First copy only the dependency files
192197
COPY --from=nemo-rl pyproject.toml uv.lock ./

‎docs/about/model-support.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ for model sizes under 70B at up to 32k sequence length.
1515
**LLMs**:
1616

1717
- **Qwen**: Qwen3.5-9B/35B-A3B/397B-A17B, Qwen3-1.5B/8B/32B, Qwen3-30B-A3B, Qwen3-235B-A22B, Qwen2.5-1.5B/7B/32B
18-
- **GLM**: GLM-4.7-Flash, GLM-5.1
18+
- **GLM**: GLM-4.7-Flash, GLM-5.1, GLM-5.2
1919
- **Llama**: Llama 3.1/3.3-8B, Llama 3.1/3.3-70B, Llama 3.2-1B
2020
- **DeepSeek**: DeepSeek-V3/R1-671B
2121
- **Mistral**: Mistral-NeMo-12B

‎docs/about/performance-summary.md‎

Lines changed: 34 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
As part of the NVIDIA NeMo Framework, NeMo RL provides optimal performance for reinforcement learning on generative AI models by incorporating the latest optimizations - such as refit optimizations, mixed-precision training, and off-policy training.
55

6-
This page provides performance benchmarks for LLMs and VLMs using NeMo RL across different GPU systems and configurations. The recipes to reproduce these runs, in yaml file form, can be found under [this folder](https://github.com/NVIDIA-NeMo/RL/tree/r0.6.0/examples/configs/recipes/llm/performance).
6+
This page provides performance benchmarks for LLMs and VLMs using NeMo RL across different GPU systems and configurations. The recipes to reproduce these runs, in yaml file form, can be found under [this folder](https://github.com/NVIDIA-NeMo/RL/tree/v0.7.0/examples/configs/recipes/llm/performance).
77

88
## Nomenclature
99

@@ -12,10 +12,12 @@ This page provides performance benchmarks for LLMs and VLMs using NeMo RL across
1212
- **TP**: Tensor Parallel Size
1313
- **PP**: Pipeline Parallel Size
1414
- **CP**: Context Parallel Size
15-
- **VP**: Virtual Pipeline Parallel Size
15+
- **VPP**: Virtual Pipeline Parallel Size
1616
- **EP**: Expert Parallel Size
1717
- **T-**: Training related
1818
- **G-**: Generation related
19+
- **T-GBS**: Training global batch size (`policy.train_global_batch_size`)
20+
- **G-GBS**: Number of rollout samples generated per step (`grpo.num_prompts_per_step * grpo.num_generations_per_prompt`)
1921
- **Training backend**: NeMo RL has two training backends: Megatron and PyTorch DTensor. This performance summary currently only shows numbers from the Megatron backend.
2022

2123
## Performance Metrics
@@ -33,7 +35,7 @@ Since reinforcement learning consists of training, generation and transition bet
3335

3436
## Performance Summary for Large Language Models
3537

36-
Below are performance benchmarks for various large language models organized by release version. These results were obtained using performance recipes available [here](https://github.com/NVIDIA-NeMo/RL/tree/r0.6.0/examples/configs/recipes/llm/performance).
38+
Below are performance benchmarks for various large language models organized by release version. These results were obtained using performance recipes available [here](https://github.com/NVIDIA-NeMo/RL/tree/v0.7.0/examples/configs/recipes/llm/performance).
3739

3840
The performance data includes:
3941

@@ -43,26 +45,30 @@ The performance data includes:
4345

4446
---
4547

46-
## NeMo RL v0.6
48+
## NeMo RL v0.7
4749

4850
### H100 BF16 Benchmarks
49-
* GRPO Dataset: [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2); DAPO dataset: [DAPOMath17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k); SWE dataset: refer to [Nemotron super-v3 documentation - stage 2.2](https://github.com/NVIDIA-NeMo/RL/blob/super-v3/docs/guides/nemotron-3-super.md#stage-22---swe-2-64-nodes)
51+
* GRPO Dataset: [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2); DAPO dataset: [DAPOMath17k](https://huggingface.co/datasets/BytedTsinghua-SIA/DAPO-Math-17k)
5052
* System: DGX-H100
5153
* Precision: Training BF16, Generation BF16
5254
* Training Backend: Megatron Core.
5355

5456
| Algorithm | Model |On/Off policy|T-Max Sequence Length|G-Average Seq len|#-GPUs|G-GBS|T-GBS|Generation [TP,PP]|Training [TP,CP,EP,PP,VPP]|Tokens / sec / GPU|Total Step time(s)|
5557
|--------- |------- |-------- |----- |----- |------|---- |---- |---- |---- |--- |---|
56-
| GRPO |DeepSeek V3|On policy |1,536 |701 |256 |512 |512 |[32,1] |[1,1,16,16,n/a] |12.1 | 134|
57-
| GRPO |DeepSeek V3|On policy |1,536 |697 |512 |512 |512 |[32,1] |[1,1,16,16,n/a] |7.24 | 111|
58-
| GRPO |DeepSeek V3|1-step Off |1,536 |710 |512 |512 |512 |[32,1] |[1,1,16,16,n/a] |12.8 | 64.1|
59-
| GRPO |Qwen3-235B |On policy |8,192 |5,698 |128 |512 |512 |[16,1] |[2,2,16,8,n/a] |58.9 | 395|
60-
| GRPO |Qwen3-235B |On policy |8,192 |5,713 |256 |512 |512 |[16,1] |[2,2,16,8,n/a] |37.4 | 312|
61-
| GRPO |Qwen3-235B |1-step Off |8,192 |5,721 |256 |512 |512 |[8,1] |[4,1,16,8,n/a] |58.7 | 231|
62-
| GRPO |Qwen3-30B3A|On policy |4,096 |3,203 |32 |2,048|512 |[2,1] |[1,1,8,1,n/a] |1102 | 192|
63-
| GRPO |Qwen3-30B3A|1-step Off |4,096 |3,201 |32 |2,048|512 |[2,1] |[1,1,8,2,n/a] |1414 | 152|
64-
| GRPO |Qwen3-30B3A|8-step Off |4,096 |3,206 |192 |2,048|512 |[2,1] |[1,1,8,1,n/a] |1025 | 34.5|
65-
| SWE |Nemotron-3-Nano-30B-A3B|1-step Off |131,072 |31,599 |128 |512 |512 |[8,1] |[8,8,8,1,n/a] |37.5 | 430|
58+
| GRPO | DeepSeek V3 | On policy | 1,536 | 722 | 256 | 512 | 512 | [32,1] | [1,1,16,16,n/a] | 17.8 | 93 |
59+
| GRPO | DeepSeek V3 | On policy | 1,536 | 724 | 512 | 512 | 512 | [32,1] | [1,1,32,16,n/a] | 10.2 | 81.2 |
60+
| GRPO | DeepSeek V3 | 1-step Off | 1,536 | 729 | 512 | 512 | 512 | [32,1] | [1,1,16,16,n/a] | 18 | 46.6 |
61+
| GRPO | Qwen3-235B-A22B | On policy | 8,192 | 5,719 | 128 | 512 | 512 | [16,1] | [2,2,16,8,n/a] | 63.7 | 367 |
62+
| GRPO | Qwen3-235B-A22B | On policy | 8,192 | 5,698 | 256 | 512 | 512 | [16,1] | [2,2,16,8,n/a] | 40.8 | 285 |
63+
| GRPO | Qwen3-235B-A22B | 1-step Off | 8,192 | 5,695 | 256 | 512 | 512 | [8,1] | [4,1,16,8,n/a] | 58.3 | 204 |
64+
| GRPO | Qwen3-30B-A3B | On policy | 4,096 | 3,198 | 32 | 2,048 | 2,048 | [2,1] | [1,1,8,1,n/a] | 1,232 | 171 |
65+
| GRPO | Qwen3-30B-A3B | 1-step Off | 4,096 | 3,203 | 32 | 2,048 | 2,048 | [2,1] | [1,1,8,2,n/a] | 1,522 | 141 |
66+
| GRPO | Qwen3-30B-A3B | 8-step Off | 4,096 | 3,203 | 192 | 2,048 | 512 | [2,1] | [1,1,8,1,n/a] | 1,067 | 33.1 |
67+
| GRPO | Qwen3-32B | 1-step Off | 4,096 | 3,258 | 64 | 2,048 | 2,048 | [4,1] | [4,1,1,4,n/a] | 675 | 161 |
68+
| GRPO | Qwen3-32B | On policy | 4,096 | 3,256 | 32 | 2,048 | 2,048 | [4,1] | [4,1,1,4,n/a] | 665 | 323 |
69+
| GRPO | Qwen3-30B-A3B | On policy | 40,960 | 8,170 | 32 | 2,048 | 512 | [2,1] | [4,8,8,1,n/a] | 262 | 2,022 |
70+
| GRPO | Nemotron-3-Super-120B-A12B | On policy | 8,192 | 3,197 | 256 | 256 | 256 | [8,1] | [4,1,32,1,n/a] | 30.7 | 108 |
71+
| GRPO | Nemotron-3-Super-120B-A12B | 1-step Off | 8,192 | 3,207 | 256 | 256 | 256 | [8,1] | [4,1,32,1,n/a] | 48.6 | 71.9 |
6672

6773
### H100 FP8 Benchmarks
6874
* GRPO Dataset: [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2)
@@ -72,7 +78,7 @@ The performance data includes:
7278

7379
| Algorithm | Model |On/Off policy|T-Max Sequence Length|G-Average Seq len|#-GPUs|G-GBS|T-GBS|Generation [TP,PP]|Training [TP,CP,EP,PP,VPP]|Tokens / sec / GPU|Total Step time(s)|
7480
|--------- |------- |-------- |----- |----- |------|---- |---- |---- |---- |--- |---|
75-
| GRPO |DeepSeek V3|1-step Off |1,536 |721 |512 |512 |512 |[16,1] |[1,1,16,16,n/a] |14.1 | 59.2|
81+
| GRPO | DeepSeek V3 | 1-step Off | 1,536 | 738 | 512 | 512 | 512 | [16,1] | [1,1,16,16,n/a] | 14 | 61.9 |
7682

7783
### GB200 BF16 Benchmarks
7884
* GRPO Dataset: [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2)
@@ -82,17 +88,20 @@ The performance data includes:
8288

8389
| Algorithm | Model |On/Off policy|T-Max Sequence Length|G-Average Seq len|#-GPUs|G-GBS|T-GBS|Generation [TP,PP]|Training [TP,CP,EP,PP,VPP]|Tokens / sec / GPU|Total Step time(s)|
8490
|--------- |------- |-------- |----- |----- |------|---- |---- |---- |---- |--- |---|
85-
| GRPO |DeepSeek V3|On policy |1,536 |711 |128 |512 |512 |[32,1] |[1,1,16,8,n/a] |30.2 | 108|
86-
| GRPO |DeepSeek V3|On policy |1,536 |700 |256 |512 |512 |[32,1] |[1,1,16,8,n/a] |16.4 | 98.7|
87-
| GRPO |DeepSeek V3|1-step Off |1,536 |708 |256 |512 |512 |[16,1] |[1,1,16,8,n/a] |26.7 | 61.7|
88-
| GRPO |Qwen3-235B |On policy |8,192 |5,709 |64 |512 |512 |[8,1] |[2,2,16,4,n/a] |163 | 286|
89-
| GRPO |Qwen3-235B |On policy |8,192 |5,693 |128 |512 |512 |[8,1] |[2,2,16,4,n/a] |67.4 | 345|
90-
| GRPO |Qwen3-235B |1-step Off |8,192 |5,705 |128 |512 |512 |[8,1] |[4,1,16,4,n/a] |85.5 | 278|
91-
| GRPO |Qwen3-30B3A|On policy |4,096 |3,199 |16 |2,048|512 |[1,1] |[1,1,16,1,n/a] |1,910 | 221|
92-
| GRPO |Qwen3-30B3A|1-step Off |4,096 |3,197 |16 |2,048|512 |[1,1] |[1,1,16,1,n/a] |1,406 | 301|
91+
| GRPO | DeepSeek V3 | On policy | 1,536 | 719 | 128 | 512 | 512 | [32,1] | [1,1,16,8,n/a] | 42 | 78.5 |
92+
| GRPO | DeepSeek V3 | On policy | 1,536 | 728 | 256 | 512 | 512 | [32,1] | [1,1,32,8,n/a] | 23.6 | 70.7 |
93+
| GRPO | DeepSeek V3 | 1-step Off | 1,536 | 719 | 256 | 512 | 512 | [16,1] | [1,1,16,8,n/a] | 47.9 | 35 |
94+
| GRPO | Qwen3-235B-A22B | On policy | 8,192 | 5,707 | 64 | 512 | 512 | [8,1] | [2,2,16,4,n/a] | 174 | 268 |
95+
| GRPO | Qwen3-235B-A22B | On policy | 8,192 | 5,711 | 128 | 512 | 512 | [8,1] | [2,2,16,4,n/a] | 111 | 210 |
96+
| GRPO | Qwen3-235B-A22B | 1-step Off | 8,192 | 5,699 | 128 | 512 | 512 | [8,1] | [4,1,16,4,n/a] | 146 | 160 |
97+
| GRPO | Qwen3-30B-A3B | On policy | 4,096 | 3,198 | 16 | 2,048 | 2,048 | [1,1] | [1,1,16,1,n/a] | 2,265 | 186 |
98+
| GRPO | Qwen3-30B-A3B | 1-step Off | 4,096 | 3,200 | 16 | 2,048 | 2,048 | [1,1] | [1,1,8,1,n/a] | 1,748 | 242 |
99+
| GRPO | Qwen3-32B | 1-step Off | 4,096 | 3,255 | 32 | 2,048 | 2,048 | [1,1] | [2,1,1,1,n/a] | 1,539 | 140 |
100+
| GRPO | Qwen3-32B | On policy | 4,096 | 3,254 | 16 | 2,048 | 2,048 | [2,1] | [2,1,1,4,n/a] | 1,458 | 295 |
101+
| GRPO | Nemotron-3-Super-120B-A12B | On policy | 8,192 | 3,219 | 128 | 256 | 256 | [4,1] | [2,1,16,1,n/a] | 56.5 | 119 |
102+
| GRPO | Nemotron-3-Super-120B-A12B | 1-step Off | 8,192 | 3,177 | 128 | 256 | 256 | [4,1] | [2,1,16,1,n/a] | 92.2 | 76 |
93103

94104
Note:
95105

96106
* All Mixture-of-expert (MoE) model training uses token drop-less.
97107
* The following metrics are extracted from the average of 5 steps: G-Average Seq len, Tokens/sec/gpu, Total Step time(s). Because of the averaging, the numbers in the table do not completely match the equation stated in Performance Metrics above but the difference is small.
98-
* There was a change in pretrained checkpoint (see [docs/guides/deepseek.md](https://github.com/NVIDIA-NeMo/RL/blob/r0.6.0/docs/guides/deepseek.md)) for DeepSeek V3 leading to lower Average Seq len. The reported throughput is not comparable across versions. Please use equivalent checkpoints for comparison. For example, using the newer checkpoint `DeepSeek V3 on-policy GRPO #-GPUs: 128` v0.5.0 performs at `26.1 Tokens / sec / GPU` compared to v0.6.0 at `30.2 Tokens / sec / GPU`.

‎docs/assets/glm5.2_grpo_curve.png‎

241 KB
Loading

‎docs/broken_links_false_positives.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,4 @@
55
{"uri": "https://huggingface.co/docs/transformers/en/chat_templating_writing"}
66
{"uri": "https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForCausalLM"}
77
{"uri": "https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoModelForImageTextToText"}
8+
{"uri": "https://opentelemetry.io/"}

0 commit comments

Comments
 (0)