Repository navigation
feat(single-controller): vLLM rollout-only simulator (cpu train backend) - #4914
shikicloud wants to merge 13 commits into
Conversation
Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…e data_plane.enabled Signed-off-by: Shiki Wu <shikiw@nvidia.com>
shikicloud
left a comment
There was a problem hiding this comment.
This PR adds examples/run_single_controller_simulator.py with --mode rollout_only: an SC recipe's rollout side (vLLM generation, the environments, the TransferQueue data plane) runs for real on the same pumps, sampler and refit lifecycle as a training run, while a CpuTrainer holds --train-time-s where the train step would run and a NoOpWeightSynchronizer runs the refit pause and resume for --refit-time-s without moving weights; the rollout-only smoke is wired into the L1 SC lane. The seams are clean: the trainer factory keeps one return shape for both trainers (setup.py#L1867-L1885), the no-op synchronizer implements every abstract member of the weight-sync ABC, the simulated train step is one awaited call inside the controller's existing timer (single_controller.py#L3649-L3654), simulator validation runs at the setup choke point before any engine starts (setup.py#L1196-L1197), and CpuTrainer is unit-tested against the in-process data plane with no Ray or GPU (test_simulator.py#L293).
Before marking the PR ready for review
- CI label. The PR is a draft with no
CI:*label. AddCI:L1: the new smoke is wired into the L1 SC lane (L1_Functional_Tests_SingleController_1.sh#L44-L46), functional lanes run only atCI:L1and above (cicd-main.yml#L194), and the quality-check job stays red until a label is attached (#L1130-L1141). The PR also edits the SC train loop (single_controller.py#L3864-L3872). A vetter then posts/ok to test a348e01f05d5ae57b4aabd71880b373ab8501a4f. - Rebase. A merge against main (6e68b17) conflicts in
nemo_rl/algorithms/single_controller_utils/setup.pyandtests/unit/single_controller/test_setup.py. Main commits since the base that touch those files: #4818, #4200, #4245, #4124, #3321, #4684, #4712, #3964, #4551. Others that auto-merge but touch PR files: #4919, #4812, #4730, #4473, #4728, #4769, #4859, #4857. After the rebase, re-run test_simulator.py, test_setup.py, test_entrypoint.py and the smoke. - PR template (PULL_REQUEST_TEMPLATE.md). "Issues" should read "Part of #4860 (rollout_only only; train_only and the cpu rollout backend stay open)", not "Closes", as the guide itself says (single-controller.md#L451). "Usage" is the command from the guide (#L434-L437) plus the smoke invocation. Tick the pre-checks and say which tests ran locally and on what hardware; the smoke needs GPUs (
cluster.gpus_per_node=2). #4814 is the legacy-path driver this was modeled on; it is unmerged and this PR does not depend on it. - One real run. The PR adds a measurement tool and changes no numerics, and the body carries no numbers. Please paste 3–4 steps of
timing/train/total_step_time,train/simulated_train_time_s,train/simulated_train_idle_skipped_s,timing/train/weight_sync,train/max_seq_mult_prob_error(expected exactly 1.0) andtrain/reward, with the--train-time-sand--refit-time-sused. Keys: single_controller.py#L3869-L3870 under thetrain/prefix (#L3939-L3941); the step timer (#L3050) and thetiming/trainprefix (#L3948-L3952); theweight_synctimer (#L3819). That lets a reader check the guide's claim that the e2e step time istotal_step_time + simulated_train_idle_skipped_s(single-controller.md#L444) and the quantities the smoke gates on (#L117-L121).
Rebase notes (not blockers)
- #4975 made
grpo.max_num_epochs: nullmean "no epoch limit" (grpo.py#L360 on main) and main's SC loop honours None.apply_simulator_overridesturns an explicit null intonum_stepsepochs. Harmless, but the comment "The recipe may omit max_num_epochs and take the schema default" goes stale, and"max_num_epochs" in algowould tell an omitted key from an explicit null (see C20). - #4200 adds SC checkpoint and resume on the paths C3 touches; the
checkpointing.enabled=falserejection must survive the merge. - On main, setup reads
trainer.worker_groupandtrainer.sharding_annotationswhentoken_capture.forest_trainingis on (setup.py#L2280-L2285 on main, #L2311-L2318). After the rebase a forest-training recipe dies in setup onCpuTrainer.worker_groupwith "CpuTrainer has no worker group" and no hint why;validate_simulator_configshould rejecttoken_capture.forest_trainingas a trainer-side feature. Main's SC driver also callsmaterialize_vllm_video_config(config.policy, config.data)after the MasterConfig build (run_grpo_single_controller.py#L112 on main); the simulator driver should mirror it.
Review environment
Seven agents reviewed head a348e01. The review host had no gh or GitHub token, no GPU, and the repo's Python pin (3.13.14) could not be installed, so no test was executed: every suggested test is marked "not verified". The pinned ruff hooks (uvx ruff@0.9.9 check, isort and format) pass on all 10 PR Python files; pre-commit's pyrefly step and CI's zero-error whitelist check could not be run locally. No CI has run on this draft.
Inline comments
Medium: C1 is_eval=True also turns on vLLM detokenization, so the measured engine is not the training engine.
Low: C2 reject vllm_cfg.load_format: dummy at setup; C3 refuse to resume from a populated checkpoint directory; C4 the guide's gen_kl_error sentence; C5 the guide's "recipe above" pointer; C6 the guide's "every node serves generation" sentence; C7 the observability docs' launcher lists; C8 derive the mode and backend tuples with get_args; C9 import the logprob column names from schema.py; C10 drop train_backend; C11 and C12 gate the simulated train step and its metrics on uses_cpu_trainer; C13 share the inference-sizing block with _build_clusters; C14 the train_fields annotation, with a trainer Protocol as a follow-up; C23 and C24 two more signature drifts from TQPolicy.
Tests: C15 and C16 a main() fixture with two tests and its import; C17 a ragged-rows copy test; C18 a smoke gate that can fail; C19 drop the dead validation dataset from the smoke; C20 the omitted-epoch branch test; C21 pump-level tests for the simulated train step; C22 a test for C2's guard.
Devil's advocate pass: 24 comments posted after 5 drops or disputes (the getattr on the simulator field, the is_trajectory_collection pop, the shard-death log line, the moved call-site defaults, --train-time-s guidance) and 3 downgrades (the idle-node warning became a docs sentence, the trainer Protocol became a follow-up, the observability docs ask was trimmed to two targets).
Generated by Claude Code
…nly-simulator Signed-off-by: Shiki Wu <shikiw@nvidia.com> # Conflicts: # nemo_rl/algorithms/single_controller_utils/setup.py # tests/unit/single_controller/test_setup.py
Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…imulator driver Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…nly-simulator Signed-off-by: Shiki Wu <shikiw@nvidia.com> # Conflicts: # nemo_rl/algorithms/single_controller.py # nemo_rl/algorithms/single_controller_utils/setup.py
…4/C23/C24) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Shiki Wu <shikiw@nvidia.com>
shuyixiong
left a comment
There was a problem hiding this comment.
One docstring correction on the simulator hold (C28).
Generated by Claude Code
shuyixiong
left a comment
There was a problem hiding this comment.
A non-blocking follow-up on driver and controller placement in simulator runs (C32).
Generated by Claude Code
… docstrings and the head-node caveat in the guide Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…cover non-Megatron and load failure - Submit build_tasks["trainer"] for the held-back trainer and for the rebuild, so a replacement task (open PR #4914's stand-in) is not skipped. - The "overlapped with ..." log line no longer claims the trainer overlapped the load. - Comments follow the repo rule: no timing on the constant, no "read lazily" claim, one line inside the function body. - Tests: the overlap test now uses a wait longer than the load's own wait, so it fails if a non-Megatron trainer is ever held back; a new test covers a dataset load that fails while a Megatron trainer waits (the trainer is never built and Gym is shut down). Signed-off-by: sniverty <sniverty@nvidia.com>
…nly-simulator Signed-off-by: Shiki Wu <shikiw@nvidia.com>
… merging main Signed-off-by: Shiki Wu <shikiw@nvidia.com>
yuki-97
left a comment
There was a problem hiding this comment.
Partial review: nemo_rl/algorithms/single_controller_utils/ and nemo_rl/algorithms/single_controller.py have not been reviewed yet.
Thanks for adding rollout-only profiling on the SingleController path while keeping real generation and environments. The GB200 comparisons and curves are helpful, and thanks for the thorough self-review.
Scope: Entrypoints, documentation and functional-test checks at 1749900. Review of the controller implementation and its utility modules is deferred.
Remaining comments
- Rebase on main and preserve dedicated-head controller placement in the simulator launcher.
- Keep simulator settings in a separate YAML loaded alongside the training recipe.
- Move the simulator section into
docs/guides/single-controller-simulator.md, register it in the docs index, and leave a short link in the main guide. - Correct the documented router placement, conditional logprob metric and simulator-only forest restriction.
nit
- Share the entrypoint execution pipeline through
single_controller_utils/entrypoint_utils.pyand move watchdog configuration and liveness helpers intosingle_controller_utils/watchdog.py. - Add simulator entrypoint wiring coverage and strengthen the functional check to require six steps and the configured refit delay.
- Shorten the two long comment blocks in the simulator entrypoint.
Validation: Static inspection only; unit, functional and GPU tests were not run in this pass.
Generated by Codex (#4999 version)
| return SimulatorSettings(**given) | ||
|
|
||
|
|
||
| def main() -> None: |
There was a problem hiding this comment.
nit: The configuration, Ray startup, setup, launch and cleanup pipeline in run_single_controller_simulator.py:165–293 copies run_grpo_single_controller.py:80–245. Simulator-specific work is concentrated around overrides and run_single_controller_simulator.py:242–265, so shared startup changes still require parallel edits.
Suggest fix:
- Extract
run_single_controller(...)intonemo_rl/algorithms/single_controller_utils/entrypoint_utils.pyand call it from both CLI wrappers, keeping flag parsing separate and preserving training's derived refit flags and cleanup inrun_grpo_single_controller.py:213–245. - Update the entrypoint tests to exercise both wrappers and their mode-specific arguments.
There was a problem hiding this comment.
Done in 46d7324. run_single_controller(config_path, overrides, *, simulator_config_path=None) holds the pipeline (config load, Ray startup, setup, launch, liveness watch, cleanup) and both scripts only parse their flags. Training keeps its derived refit flags and the teacher / value cleanup, and its comments moved unchanged from run_grpo_single_controller.py. Three functional hooks and two source-scanning unit tests that read the old launcher now point at the shared module.
| ) | ||
|
|
||
| print("🚀 Launching SingleControllerActor (simulator)") | ||
| sc = SingleControllerActor.remote( |
There was a problem hiding this comment.
When rebasing, this launcher also needs main's head-placement behavior: run_grpo_single_controller.py:81–89 selects the dedicated-head resource and run_grpo_single_controller.py:217–225 applies it, while this entrypoint still launches without those options.
Suggest fix:
- Share the placement helper through
utils.py:458–502and use it in the launch path reached by both entrypoints; updatesingle-controller.md:600andcluster.md:143to describe both commands. - Preserve and adapt main's placement tests for the shared launch path.
There was a problem hiding this comment.
Done in 46d7324. main is merged twice in this push (5939fa5 before the review changes, 99ae93a after them), so the branch has no conflicts with main as of 2026-11-16. Both launchers now go through run_single_controller in single_controller_utils/entrypoint_utils.py, which applies controller_placement() (main's _controller_placement, moved) to SingleControllerActor.options(...). main's placement tests are kept in test_entrypoint.py, against the shared path and parametrized over both launchers. docs/cluster.md names both commands, and the simulator guide's Placement section says the controller is placed as in a training run.
| Compaction, mixed image/video conversations, native audio, | ||
| video token pruning, static tiling (`image_num_patches`), other processor families, | ||
| and `token_capture.defer_routed_experts_to_policy: true` are not supported. | ||
| and `token_capture.defer_routed_experts_to_policy: true` / `forest_training: true` are not supported. |
There was a problem hiding this comment.
forest_training is supported with captured media: rollout_reassembler.py:500–502 explicitly fetches forest media, and test_captured_media.py:1581–1658 checks a successful shared-image forest. This general VLM limitation incorrectly applies the simulator-only restriction to training.
Suggest fix: remove only the forest entry from this VLM bullet.
| and `token_capture.defer_routed_experts_to_policy: true` / `forest_training: true` are not supported. | |
| and `token_capture.defer_routed_experts_to_policy: true` are not supported. |
There was a problem hiding this comment.
Fixed in 46d7324: the VLM bullet is main's text again. The simulator-only restriction stays in the simulator guide's Limitations.
| What the stand-ins do, per step: | ||
|
|
||
| - The rollout pump, `RolloutManager`, `TQReplayBuffer` and the configured sampler run unchanged, so admission is gated exactly as in training: on the trainer version, or on buffer capacity for the `windowed` sampler. | ||
| - The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`. |
There was a problem hiding this comment.
nit: train/max_seq_mult_prob_error is absent when the supported policy-logprob skip condition in utils.py:24–35 applies; advantage_stage.py:482–486 only records it when that stage runs.
Suggest fix: qualify the copied columns and diagnostic by which logprob stages are required.
| - The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`. | |
| - The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into each required policy/reference column. When the policy-logprob stage runs, `train/max_seq_mult_prob_error` reads 1.0; when that stage is skipped, the metric is absent. After all chunks are accepted, it holds `--train-time-s` once. Generation keeps serving during this delay, but forward/backward work overlapping later rollout chunks is not modeled. The delay ends early when rollout activity is idle; adding `train/simulated_train_idle_skipped_s` to `total_step_time` reconstructs this coarse timing model. |
There was a problem hiding this comment.
Fixed in 46d7324. The guide now says the CpuTrainer fills each logprob column the recipe requires, that train/max_seq_mult_prob_error reads 1.0 when the policy-logprob stage runs and is absent when it is skipped, and that the hold runs once after all chunks are accepted, so overlap with later chunks of the same step is not modeled.
| uv run tests/check_metrics.py $JSON_METRICS \ | ||
| 'max(data["train/reward"]) > 0' \ | ||
| 'abs(max(data["train/max_seq_mult_prob_error"]) - 1.0) < 1e-6' \ | ||
| 'min(data["train/simulated_train_time_s"]) >= 0' \ | ||
| 'max(data["train/simulated_train_idle_skipped_s"]) <= 15' |
There was a problem hiding this comment.
nit: these checks accept a one-step run and never inspect the simulated refit delay, despite the six-step, three-second configuration in grpo_async_gym_single_controller_rollout_only.sh:60–62. The fixed sleep in simulator.py:345–350 runs inside single_controller.py:3578–3589.
Suggest fix: also require all six step metrics and a mean weight-sync time of at least three seconds. Proposed replacement for the metric-check command, not run:
uv run tests/check_metrics.py $JSON_METRICS \
'max(data["train/reward"]) > 0' \
'abs(max(data["train/max_seq_mult_prob_error"]) - 1.0) < 1e-6' \
'min(data["train/simulated_train_time_s"]) >= 0' \
'max(data["train/simulated_train_idle_skipped_s"]) <= 15' \
'len(data["train/simulated_train_time_s"]) == 6' \
'mean(data["timing/train/weight_sync"]) >= 3'There was a problem hiding this comment.
Done in 46d7324, both checks added. Run on GB200 with vLLM 0.33.0 on the pushed tree: 6 steps, mean timing/train/weight_sync 3.03 s, all six checks pass.
|
|
||
| ## Rollout-Only Simulator | ||
|
|
||
| `examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run. |
There was a problem hiding this comment.
nit: the router has an explicit placement rule: setup.py:1102–1106 hard-pins it to the driver's node. This paragraph incorrectly groups it with processes that have no such rule.
Suggest fix: state the router affinity and avoid promising that the unconstrained controller always runs on the head.
| `examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run. | |
| `examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver and possibly the `SingleControllerActor`; the generation router, when enabled, is pinned to the driver's node. On a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run. |
There was a problem hiding this comment.
Fixed in 46d7324, in the guide's Placement section: the router is pinned to the driver's node; the controller is pinned to the dedicated head when the cluster has one and is otherwise unconstrained.
| # Configured as run_grpo_single_controller.py configures it (same | ||
| # skip_tokenizer_init rule, so the engine does the same per-token | ||
| # work as one beside a trainer), not as an eval engine. load_format | ||
| # is already "auto" from apply_simulator_overrides, and no refit | ||
| # delivers draft or MTP weights, so none are expected from one. |
There was a problem hiding this comment.
nit: two added comment blocks inside functions exceed the two-line limit in the in-function comment rule.
Suggest fix: shorten both blocks to one line; use these replacements together.
run_single_controller_simulator.py:215–217 replacement:
# Export telemetry settings before Ray snapshots the worker environment.| # Configured as run_grpo_single_controller.py configures it (same | |
| # skip_tokenizer_init rule, so the engine does the same per-token | |
| # work as one beside a trainer), not as an eval engine. load_format | |
| # is already "auto" from apply_simulator_overrides, and no refit | |
| # delivers draft or MTP weights, so none are expected from one. | |
| # Keep training's tokenizer setup; the engine loads its own checkpoint. |
There was a problem hiding this comment.
Done in 46d7324: the launcher is a thin wrapper now, and the two simulator-specific comments in entrypoint_utils._configure_generation are one line each.
| ## Rollout-Only Simulator | ||
|
|
||
| `examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run. | ||
|
|
||
| ```bash | ||
| uv run examples/run_single_controller_simulator.py --config <your-sc.yaml> \ | ||
| --num-steps 20 --num-nodes 4 --train-time-s 300 --refit-time-s 20 \ | ||
| <hydra overrides...> | ||
| ``` | ||
|
|
||
| The recipe is used as-is and must be an SC recipe (`data_plane.enabled: true`, `async_rl:`, `grpo.async_grpo: null`). The driver overrides only what a rollout-only run owns: `--num-steps` sets `max_num_steps` (and widens `max_num_epochs` so a short dataset cannot clamp it), `--num-nodes` sets the generation node count, `data.validation` is dropped, and `vllm_cfg.load_format` defaults to `auto` so the engine loads the real checkpoint instead of dummy weights plus a first refit (an explicit `dummy` is rejected). Everything else about the engine is configured exactly as `run_grpo_single_controller.py` configures it, including the `skip_tokenizer_init` rule, so it does the same per-token work as an engine beside a trainer. | ||
|
|
||
| What the stand-ins do, per step: | ||
|
|
||
| - The rollout pump, `RolloutManager`, `TQReplayBuffer` and the configured sampler run unchanged, so admission is gated exactly as in training: on the trainer version, or on buffer capacity for the `windowed` sampler. | ||
| - The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`. | ||
| - `_sync_weights` runs its full lifecycle -- dispatch gate, `pause_generation_for_refit(clear_cache=recompute_kv_cache_after_weight_updates)`, resume -- around a `NoOpWeightSynchronizer` that moves nothing and holds `--refit-time-s`. The trainer version advances, which releases the sampler's next target. | ||
|
|
||
| Everything on the rollout side is available: `generation_router`, `generation_fleet_health` (probes and shard restarts), the stall watchdog, token capture, and the `NRL_NSYS_*` profiling hooks. | ||
|
|
||
| `tests/functional/grpo_async_gym_single_controller_rollout_only.sh` is the smoke for this path: the recipe and overrides of `tests/functional/grpo_async_gym_single_controller.sh` (`examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml` on Qwen3-0.6B with a 1-GPU vLLM split) with the trainer stubbed out, gated on a scored rollout and on the stand-in's metrics. | ||
|
|
||
| Not supported by `rollout_only`: PPO, non-vLLM generation backends (Megatron generation serves the trainer's own weights; TRT-LLM and SGLang are not validated under the SC refit lifecycle), Dynamo serving (`policy.generation.dynamo: true`, not validated under the simulator's refit lifecycle yet), checkpointing, on-policy distillation, `token_capture.defer_routed_experts_to_policy`, `token_capture.forest_training`, refittable fp8 KV caches, adaptive-ppo-clip, and online quantization (`precision: fp8`, `nvfp4_pertoken_rollout`, `quant_cfg`, `real_quant`): those quantize at refit, which this run has none of. Setup rejects each before any engine starts. The `train_only` mode and the `cpu` rollout backend are tracked in [issue #4860](https://github.com/NVIDIA-NeMo/RL/issues/4860). |
There was a problem hiding this comment.
The simulator has a separate entrypoint, configuration overrides and timing model; a dedicated guide would keep this page focused on the main SingleController workflow.
Suggest fix: move the new single-controller.md:598–620 section into docs/guides/single-controller-simulator.md, carrying the requested simulator corrections with it. Then leave the short link below and register the guide in docs/index.md:342.
| ## Rollout-Only Simulator | |
| `examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run. | |
| ```bash | |
| uv run examples/run_single_controller_simulator.py --config <your-sc.yaml> \ | |
| --num-steps 20 --num-nodes 4 --train-time-s 300 --refit-time-s 20 \ | |
| <hydra overrides...> | |
| ``` | |
| The recipe is used as-is and must be an SC recipe (`data_plane.enabled: true`, `async_rl:`, `grpo.async_grpo: null`). The driver overrides only what a rollout-only run owns: `--num-steps` sets `max_num_steps` (and widens `max_num_epochs` so a short dataset cannot clamp it), `--num-nodes` sets the generation node count, `data.validation` is dropped, and `vllm_cfg.load_format` defaults to `auto` so the engine loads the real checkpoint instead of dummy weights plus a first refit (an explicit `dummy` is rejected). Everything else about the engine is configured exactly as `run_grpo_single_controller.py` configures it, including the `skip_tokenizer_init` rule, so it does the same per-token work as an engine beside a trainer. | |
| What the stand-ins do, per step: | |
| - The rollout pump, `RolloutManager`, `TQReplayBuffer` and the configured sampler run unchanged, so admission is gated exactly as in training: on the trainer version, or on buffer capacity for the `windowed` sampler. | |
| - The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`. | |
| - `_sync_weights` runs its full lifecycle -- dispatch gate, `pause_generation_for_refit(clear_cache=recompute_kv_cache_after_weight_updates)`, resume -- around a `NoOpWeightSynchronizer` that moves nothing and holds `--refit-time-s`. The trainer version advances, which releases the sampler's next target. | |
| Everything on the rollout side is available: `generation_router`, `generation_fleet_health` (probes and shard restarts), the stall watchdog, token capture, and the `NRL_NSYS_*` profiling hooks. | |
| `tests/functional/grpo_async_gym_single_controller_rollout_only.sh` is the smoke for this path: the recipe and overrides of `tests/functional/grpo_async_gym_single_controller.sh` (`examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml` on Qwen3-0.6B with a 1-GPU vLLM split) with the trainer stubbed out, gated on a scored rollout and on the stand-in's metrics. | |
| Not supported by `rollout_only`: PPO, non-vLLM generation backends (Megatron generation serves the trainer's own weights; TRT-LLM and SGLang are not validated under the SC refit lifecycle), Dynamo serving (`policy.generation.dynamo: true`, not validated under the simulator's refit lifecycle yet), checkpointing, on-policy distillation, `token_capture.defer_routed_experts_to_policy`, `token_capture.forest_training`, refittable fp8 KV caches, adaptive-ppo-clip, and online quantization (`precision: fp8`, `nvfp4_pertoken_rollout`, `quant_cfg`, `real_quant`): those quantize at refit, which this run has none of. Setup rejects each before any engine starts. The `train_only` mode and the `cpu` rollout backend are tracked in [issue #4860](https://github.com/NVIDIA-NeMo/RL/issues/4860). | |
| ## Rollout-Only Simulator | |
| See the [SingleController simulator guide](single-controller-simulator.md) for setup, usage, timing metrics and limitations. |
docs/index.md:342 replacement:
guides/single-controller.md
guides/single-controller-simulator.md
There was a problem hiding this comment.
Done in 46d7324: the section moved to docs/guides/single-controller-simulator.md, registered in docs/index.md, with the corrections from the other threads. The main guide keeps the short link.
| setup_single_controller, | ||
| ) | ||
| from nemo_rl.algorithms.single_controller_utils.utils import ( | ||
| run_with_controller_liveness_watch, |
There was a problem hiding this comment.
nit: could you help to add a new nemo_rl/algorithms/single_controller_utils/watchdog.py, and move WatchdogConfig and run_with_controller_liveness_watch to there instead of mix things in utils.py?
There was a problem hiding this comment.
Done in 46d7324: WatchdogConfig and run_with_controller_liveness_watch are in single_controller_utils/watchdog.py, and utils.py is back to main's content.
| _DEFAULTS = SimulatorSettings() | ||
|
|
||
|
|
||
| def parse_args() -> tuple[argparse.Namespace, list[str]]: |
There was a problem hiding this comment.
can we move it to a yaml to follow the existing behavior in the repo?
for the simulator, I think we can load two config yaml, one is the simulator yaml to keep new things in simulator, another is the train config just like before. since we might need to add more configs later, I think it's not clear keeping the configs just in examples/run_single_controller_simulator.py, yaml is more straightforward to show.
@terrykong wdyt?
There was a problem hiding this comment.
Done in 46d7324, in the shape you describe. --config is the SC recipe, loaded as before. --simulator-config (default examples/configs/simulator/rollout_only.yaml) holds the simulator keys and is validated by SimulatorConfig, a pydantic model with extra="forbid" that replaces the SimulatorSettings dataclass and the six flags. Overrides keep the Hydra syntax: simulator.<key>=<value> addresses the simulator YAML, anything else the recipe. test_config_validation.py checks the shipped YAML against the schema. Happy to change the shape if terrykong prefers another one.
…nly-simulator Signed-off-by: Shiki Wu <shikiw@nvidia.com> # Conflicts: # docs/guides/single-controller.md # examples/run_grpo_single_controller.py # nemo_rl/algorithms/single_controller_utils/utils.py # tests/unit/single_controller/test_entrypoint.py
… settings to YAML Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…nly-simulator Signed-off-by: Shiki Wu <shikiw@nvidia.com>
What does this PR do ?
Implement vLLM as the rollout backend of a single-controller simulator (rollout-only mode) for #4860, inspired by #4814.
examples/run_single_controller_simulator.pyruns an SC recipe's rollouts exactly as training would, with a GPU-free trainer stand-in (mode: rollout_only). It is the SingleController counterpart of #4814 (legacy path, TRT-LLM), built on the SC pumps, TransferQueue data plane, sampler and refit lifecycle.The launcher takes two YAML files:
--configis the SC recipe, used as-is, and--simulator-config(defaultexamples/configs/simulator/rollout_only.yaml) holds the simulator keys. In the recipe the launcher overrides onlymax_num_stepsand the node count (from the simulator config'snum_steps/num_nodes),data.validationandvllm_cfg.load_format(defaults toautoso the engine loads the real checkpoint;dummyis rejected). The engine is configured exactly as the training entrypoint configures it, so it does the same per-token work as an engine beside a trainer.Design
CpuTrainerreplacesTQPolicy: bootstraps TQ, GPU phase transitions are no-ops, the logprob stages copygeneration_logprobsintoprev_logprobs/reference_policy_logprobs(sotrain/max_seq_mult_prob_errorreads exactly 1.0), the train step returns no metrics.NoOpWeightSynchronizerreplaces the transport:is_staleis False,sync_weightsholdsrefit_time_sinside the generation pause so the refit bubble can be sized like a real transfer.setup_single_controller(simulator=...): every node is an inference node, no train cluster, nocreate_weight_synchronizer.SingleControllerActor._simulate_training: holdstrain_time_swherepolicy.train()would run, ends early once no rollout is in flight, logstrain/simulated_train_time_sandtrain/simulated_train_idle_skipped_s. e2e step time =total_step_time + idle_skipped._sync_weights,generation_router,generation_fleet_health, stall watchdog and token capture are untouched and stay available.SimulatorConfig(pydantic,extra="forbid") is the schema of the simulator YAML;simulator.<key>=<value>on the command line overrides it.SingleControllerActorArgs.simulatordefaults toNone, so training runs are unchanged.run_single_controllerinsingle_controller_utils/entrypoint_utils.py(config load, Ray startup, setup, dedicated-head placement of the controller, liveness watch, cleanup).WatchdogConfigand the liveness watch live insingle_controller_utils/watchdog.py.docs/guides/single-controller-simulator.md.Rejected at setup: PPO, non-vLLM generation backends, Dynamo serving, checkpointing, OPD,
token_capture.defer_routed_experts_to_policy,token_capture.forest_training, refittable fp8 KV caches, adaptive-ppo-clip, and online quantization.Issues
Part of #4860 (rollout_only only; train_only and the cpu rollout backend stay open).
Usage
uv run python examples/run_single_controller_simulator.py \ --config examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml \ --simulator-config examples/configs/simulator/rollout_only.yaml \ simulator.num_steps=6 simulator.num_nodes=1 \ simulator.train_time_s=15 simulator.refit_time_s=3 \ policy.model_name=Qwen/Qwen3-0.6B # any SC recipe; recipe overrides pass through # CI smoke (L1 SingleController lane): bash tests/functional/grpo_async_gym_single_controller_rollout_only.shMetrics to watch:
train/simulated_train_time_s,train/simulated_train_idle_skipped_s,timing/train/weight_sync,timing/train/exposed_generation;train/max_seq_mult_prob_errorreads exactly 1.0 (it is absent when the recipe skips the policy-logprob stage).Experiment results
GB200, one node, vLLM 0.33.0, 8 steps, recipe and overrides of
tests/functional/grpo_async_gym_single_controller.sh(Qwen3-0.6B, NeMo-Gym workplace_assistant, 4 prompts x 2 generations, 1 vLLM GPU): the real SC training run (Megatron TP1 on the second GPU) and two rollout-only runs of the same recipe.train_time_s=15train_time_s=30timing/train/exposed_generations, mean of steps 2-8timing/train/total_step_times, mean of steps 2-8timing/train/weight_syncs, meantiming/train/policy_trainings, mean of steps 2-8train/simulated_train_idle_skipped_s, mean of steps 2-8train/max_seq_mult_prob_error, maxtrain/reward, mean of the 8 stepsThe rollout side matches the training run: exposed generation per step agrees (0.37 vs 0.39 s) and the per-step rewards are identical. The stand-ins show where they should:
weight_syncis the 3 s hold,policy_trainingis the part of the train hold that overlapped generation, the rest is idle-skipped. With lookahead 1 the next step generates during the hold, so exposed generation drops to ~0.Curves
Before your PR is "Ready for review"
Pre checks: