Skip to content

feat(single-controller): vLLM rollout-only simulator (cpu train backend) - #4914

Open
shikicloud wants to merge 13 commits into
NVIDIA-NeMo:mainfrom
shikicloud:shikiw/sc-rollout-only-simulator
Open

shikicloud wants to merge 13 commits into
NVIDIA-NeMo:mainfrom
shikicloud:shikiw/sc-rollout-only-simulator

Conversation

@shikicloud

@shikicloud shikicloud commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do ?

Implement vLLM as the rollout backend of a single-controller simulator (rollout-only mode) for #4860, inspired by #4814.

examples/run_single_controller_simulator.py runs an SC recipe's rollouts exactly as training would, with a GPU-free trainer stand-in (mode: rollout_only). It is the SingleController counterpart of #4814 (legacy path, TRT-LLM), built on the SC pumps, TransferQueue data plane, sampler and refit lifecycle.

The launcher takes two YAML files: --config is the SC recipe, used as-is, and --simulator-config (default examples/configs/simulator/rollout_only.yaml) holds the simulator keys. In the recipe the launcher overrides only max_num_steps and the node count (from the simulator config's num_steps / num_nodes), data.validation and vllm_cfg.load_format (defaults to auto so the engine loads the real checkpoint; dummy is rejected). The engine is configured exactly as the training entrypoint configures it, so it does the same per-token work as an engine beside a trainer.

Design

  • CpuTrainer replaces TQPolicy: bootstraps TQ, GPU phase transitions are no-ops, the logprob stages copy generation_logprobs into prev_logprobs / reference_policy_logprobs (so train/max_seq_mult_prob_error reads exactly 1.0), the train step returns no metrics.
  • NoOpWeightSynchronizer replaces the transport: is_stale is False, sync_weights holds refit_time_s inside the generation pause so the refit bubble can be sized like a real transfer.
  • setup_single_controller(simulator=...): every node is an inference node, no train cluster, no create_weight_synchronizer.
  • SingleControllerActor._simulate_training: holds train_time_s where policy.train() would run, ends early once no rollout is in flight, logs train/simulated_train_time_s and train/simulated_train_idle_skipped_s. e2e step time = total_step_time + idle_skipped.
  • _sync_weights, generation_router, generation_fleet_health, stall watchdog and token capture are untouched and stay available.
  • SimulatorConfig (pydantic, extra="forbid") is the schema of the simulator YAML; simulator.<key>=<value> on the command line overrides it. SingleControllerActorArgs.simulator defaults to None, so training runs are unchanged.
  • Both launchers are thin wrappers over run_single_controller in single_controller_utils/entrypoint_utils.py (config load, Ray startup, setup, dedicated-head placement of the controller, liveness watch, cleanup). WatchdogConfig and the liveness watch live in single_controller_utils/watchdog.py.
  • Guide: docs/guides/single-controller-simulator.md.

Rejected at setup: PPO, non-vLLM generation backends, Dynamo serving, checkpointing, OPD, token_capture.defer_routed_experts_to_policy, token_capture.forest_training, refittable fp8 KV caches, adaptive-ppo-clip, and online quantization.

Issues

Part of #4860 (rollout_only only; train_only and the cpu rollout backend stay open).

Usage

uv run python examples/run_single_controller_simulator.py \
    --config examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml \
    --simulator-config examples/configs/simulator/rollout_only.yaml \
    simulator.num_steps=6 simulator.num_nodes=1 \
    simulator.train_time_s=15 simulator.refit_time_s=3 \
    policy.model_name=Qwen/Qwen3-0.6B   # any SC recipe; recipe overrides pass through

# CI smoke (L1 SingleController lane):
bash tests/functional/grpo_async_gym_single_controller_rollout_only.sh

Metrics to watch: train/simulated_train_time_s, train/simulated_train_idle_skipped_s, timing/train/weight_sync, timing/train/exposed_generation; train/max_seq_mult_prob_error reads exactly 1.0 (it is absent when the recipe skips the policy-logprob stage).

Experiment results

GB200, one node, vLLM 0.33.0, 8 steps, recipe and overrides of tests/functional/grpo_async_gym_single_controller.sh (Qwen3-0.6B, NeMo-Gym workplace_assistant, 4 prompts x 2 generations, 1 vLLM GPU): the real SC training run (Megatron TP1 on the second GPU) and two rollout-only runs of the same recipe.

SC training rollout-only, lookahead 0, train_time_s=15 rollout-only, lookahead 1, train_time_s=30
timing/train/exposed_generation s, mean of steps 2-8 0.387 0.374 0.001
timing/train/total_step_time s, mean of steps 2-8 4.89 3.46 4.09
timing/train/weight_sync s, mean 0.074 3.028 3.028
timing/train/policy_training s, mean of steps 2-8 1.62 0.00 1.00
train/simulated_train_idle_skipped_s, mean of steps 2-8 n/a 15.0 29.0
train/max_seq_mult_prob_error, max 1.043 1.000 1.000
train/reward, mean of the 8 steps 0.188 0.188 0.188

The rollout side matches the training run: exposed generation per step agrees (0.37 vs 0.39 s) and the per-step rewards are identical. The stand-ins show where they should: weight_sync is the 3 s hold, policy_training is the part of the train hold that overlapped generation, the rest is idle-skipped. With lookahead 1 the next step generates during the hold, so exposed generation drops to ~0.

Curves

curves_4914_round4

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Oct 26, 2026
…e data_plane.enabled

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

@shikicloud shikicloud left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds examples/run_single_controller_simulator.py with --mode rollout_only: an SC recipe's rollout side (vLLM generation, the environments, the TransferQueue data plane) runs for real on the same pumps, sampler and refit lifecycle as a training run, while a CpuTrainer holds --train-time-s where the train step would run and a NoOpWeightSynchronizer runs the refit pause and resume for --refit-time-s without moving weights; the rollout-only smoke is wired into the L1 SC lane. The seams are clean: the trainer factory keeps one return shape for both trainers (setup.py#L1867-L1885), the no-op synchronizer implements every abstract member of the weight-sync ABC, the simulated train step is one awaited call inside the controller's existing timer (single_controller.py#L3649-L3654), simulator validation runs at the setup choke point before any engine starts (setup.py#L1196-L1197), and CpuTrainer is unit-tested against the in-process data plane with no Ray or GPU (test_simulator.py#L293).

Before marking the PR ready for review

  1. CI label. The PR is a draft with no CI:* label. Add CI:L1: the new smoke is wired into the L1 SC lane (L1_Functional_Tests_SingleController_1.sh#L44-L46), functional lanes run only at CI:L1 and above (cicd-main.yml#L194), and the quality-check job stays red until a label is attached (#L1130-L1141). The PR also edits the SC train loop (single_controller.py#L3864-L3872). A vetter then posts /ok to test a348e01f05d5ae57b4aabd71880b373ab8501a4f.
  2. Rebase. A merge against main (6e68b17) conflicts in nemo_rl/algorithms/single_controller_utils/setup.py and tests/unit/single_controller/test_setup.py. Main commits since the base that touch those files: #4818, #4200, #4245, #4124, #3321, #4684, #4712, #3964, #4551. Others that auto-merge but touch PR files: #4919, #4812, #4730, #4473, #4728, #4769, #4859, #4857. After the rebase, re-run test_simulator.py, test_setup.py, test_entrypoint.py and the smoke.
  3. PR template (PULL_REQUEST_TEMPLATE.md). "Issues" should read "Part of #4860 (rollout_only only; train_only and the cpu rollout backend stay open)", not "Closes", as the guide itself says (single-controller.md#L451). "Usage" is the command from the guide (#L434-L437) plus the smoke invocation. Tick the pre-checks and say which tests ran locally and on what hardware; the smoke needs GPUs (cluster.gpus_per_node=2). #4814 is the legacy-path driver this was modeled on; it is unmerged and this PR does not depend on it.
  4. One real run. The PR adds a measurement tool and changes no numerics, and the body carries no numbers. Please paste 3–4 steps of timing/train/total_step_time, train/simulated_train_time_s, train/simulated_train_idle_skipped_s, timing/train/weight_sync, train/max_seq_mult_prob_error (expected exactly 1.0) and train/reward, with the --train-time-s and --refit-time-s used. Keys: single_controller.py#L3869-L3870 under the train/ prefix (#L3939-L3941); the step timer (#L3050) and the timing/train prefix (#L3948-L3952); the weight_sync timer (#L3819). That lets a reader check the guide's claim that the e2e step time is total_step_time + simulated_train_idle_skipped_s (single-controller.md#L444) and the quantities the smoke gates on (#L117-L121).

Rebase notes (not blockers)

  • #4975 made grpo.max_num_epochs: null mean "no epoch limit" (grpo.py#L360 on main) and main's SC loop honours None. apply_simulator_overrides turns an explicit null into num_steps epochs. Harmless, but the comment "The recipe may omit max_num_epochs and take the schema default" goes stale, and "max_num_epochs" in algo would tell an omitted key from an explicit null (see C20).
  • #4200 adds SC checkpoint and resume on the paths C3 touches; the checkpointing.enabled=false rejection must survive the merge.
  • On main, setup reads trainer.worker_group and trainer.sharding_annotations when token_capture.forest_training is on (setup.py#L2280-L2285 on main, #L2311-L2318). After the rebase a forest-training recipe dies in setup on CpuTrainer.worker_group with "CpuTrainer has no worker group" and no hint why; validate_simulator_config should reject token_capture.forest_training as a trainer-side feature. Main's SC driver also calls materialize_vllm_video_config(config.policy, config.data) after the MasterConfig build (run_grpo_single_controller.py#L112 on main); the simulator driver should mirror it.

Review environment

Seven agents reviewed head a348e01. The review host had no gh or GitHub token, no GPU, and the repo's Python pin (3.13.14) could not be installed, so no test was executed: every suggested test is marked "not verified". The pinned ruff hooks (uvx ruff@0.9.9 check, isort and format) pass on all 10 PR Python files; pre-commit's pyrefly step and CI's zero-error whitelist check could not be run locally. No CI has run on this draft.

Inline comments

Medium: C1 is_eval=True also turns on vLLM detokenization, so the measured engine is not the training engine.

Low: C2 reject vllm_cfg.load_format: dummy at setup; C3 refuse to resume from a populated checkpoint directory; C4 the guide's gen_kl_error sentence; C5 the guide's "recipe above" pointer; C6 the guide's "every node serves generation" sentence; C7 the observability docs' launcher lists; C8 derive the mode and backend tuples with get_args; C9 import the logprob column names from schema.py; C10 drop train_backend; C11 and C12 gate the simulated train step and its metrics on uses_cpu_trainer; C13 share the inference-sizing block with _build_clusters; C14 the train_fields annotation, with a trainer Protocol as a follow-up; C23 and C24 two more signature drifts from TQPolicy.

Tests: C15 and C16 a main() fixture with two tests and its import; C17 a ragged-rows copy test; C18 a smoke gate that can fail; C19 drop the dead validation dataset from the smoke; C20 the omitted-epoch branch test; C21 pump-level tests for the simulated train step; C22 a test for C2's guard.

Devil's advocate pass: 24 comments posted after 5 drops or disputes (the getattr on the simulator field, the is_trajectory_collection pop, the shard-death log line, the moved call-site defaults, --train-time-s guidance) and 3 downgrades (the idle-node warning became a docs sentence, the trainer Protocol became a follow-up, the observability docs ask was trimmed to two targets).

Generated by Claude Code

Comment thread docs/guides/single-controller.md Resolved
Comment thread docs/guides/single-controller.md Resolved
Comment thread nemo_rl/algorithms/single_controller.py Resolved
shikicloud and others added 5 commits November 2, 2026 19:55
…nly-simulator

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

# Conflicts:
#	nemo_rl/algorithms/single_controller_utils/setup.py
#	tests/unit/single_controller/test_setup.py
Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…imulator driver

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…nly-simulator

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

# Conflicts:
#	nemo_rl/algorithms/single_controller.py
#	nemo_rl/algorithms/single_controller_utils/setup.py
…4/C23/C24)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Shiki Wu <shikiw@nvidia.com>
@shikicloud
shikicloud marked this pull request as ready for review November 3, 2026 04:54
@shikicloud
shikicloud requested a balanced review from Copilot November 4, 2026 05:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@shuyixiong shuyixiong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One docstring correction on the simulator hold (C28).

Generated by Claude Code

Comment thread nemo_rl/algorithms/single_controller.py Resolved

@shuyixiong shuyixiong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A non-blocking follow-up on driver and controller placement in simulator runs (C32).

Generated by Claude Code

… docstrings and the head-node caveat in the guide

Signed-off-by: Shiki Wu <shikiw@nvidia.com>
snivertynv added a commit that referenced this pull request Nov 5, 2026
…cover non-Megatron and load failure

- Submit build_tasks["trainer"] for the held-back trainer and for the rebuild, so a
  replacement task (open PR #4914's stand-in) is not skipped.
- The "overlapped with ..." log line no longer claims the trainer overlapped the load.
- Comments follow the repo rule: no timing on the constant, no "read lazily" claim, one
  line inside the function body.
- Tests: the overlap test now uses a wait longer than the load's own wait, so it fails if
  a non-Megatron trainer is ever held back; a new test covers a dataset load that fails
  while a Megatron trainer waits (the trainer is never built and Gym is shut down).

Signed-off-by: sniverty <sniverty@nvidia.com>
@terrykong terrykong added CI:L1 Run doctests, unit tests, and functional tests CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L1 Run doctests, unit tests, and functional tests labels Nov 5, 2026
…nly-simulator

Signed-off-by: Shiki Wu <shikiw@nvidia.com>
… merging main

Signed-off-by: Shiki Wu <shikiw@nvidia.com>
@shikicloud shikicloud changed the title feat(single-controller): rollout-only simulator with a cpu train backend feat(single-controller): vLLM rollout-only simulator (cpu train backend) Nov 13, 2026

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partial review: nemo_rl/algorithms/single_controller_utils/ and nemo_rl/algorithms/single_controller.py have not been reviewed yet.

Thanks for adding rollout-only profiling on the SingleController path while keeping real generation and environments. The GB200 comparisons and curves are helpful, and thanks for the thorough self-review.

Scope: Entrypoints, documentation and functional-test checks at 1749900. Review of the controller implementation and its utility modules is deferred.

Remaining comments

  • Rebase on main and preserve dedicated-head controller placement in the simulator launcher.
  • Keep simulator settings in a separate YAML loaded alongside the training recipe.
  • Move the simulator section into docs/guides/single-controller-simulator.md, register it in the docs index, and leave a short link in the main guide.
  • Correct the documented router placement, conditional logprob metric and simulator-only forest restriction.

nit

  • Share the entrypoint execution pipeline through single_controller_utils/entrypoint_utils.py and move watchdog configuration and liveness helpers into single_controller_utils/watchdog.py.
  • Add simulator entrypoint wiring coverage and strengthen the functional check to require six steps and the configured refit delay.
  • Shorten the two long comment blocks in the simulator entrypoint.

Validation: Static inspection only; unit, functional and GPU tests were not run in this pass.

Generated by Codex (#4999 version)

return SimulatorSettings(**given)


def main() -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The configuration, Ray startup, setup, launch and cleanup pipeline in run_single_controller_simulator.py:165–293 copies run_grpo_single_controller.py:80–245. Simulator-specific work is concentrated around overrides and run_single_controller_simulator.py:242–265, so shared startup changes still require parallel edits.

Suggest fix:

  1. Extract run_single_controller(...) into nemo_rl/algorithms/single_controller_utils/entrypoint_utils.py and call it from both CLI wrappers, keeping flag parsing separate and preserving training's derived refit flags and cleanup in run_grpo_single_controller.py:213–245.
  2. Update the entrypoint tests to exercise both wrappers and their mode-specific arguments.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324. run_single_controller(config_path, overrides, *, simulator_config_path=None) holds the pipeline (config load, Ray startup, setup, launch, liveness watch, cleanup) and both scripts only parse their flags. Training keeps its derived refit flags and the teacher / value cleanup, and its comments moved unchanged from run_grpo_single_controller.py. Three functional hooks and two source-scanning unit tests that read the old launcher now point at the shared module.

)

print("🚀 Launching SingleControllerActor (simulator)")
sc = SingleControllerActor.remote(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When rebasing, this launcher also needs main's head-placement behavior: run_grpo_single_controller.py:81–89 selects the dedicated-head resource and run_grpo_single_controller.py:217–225 applies it, while this entrypoint still launches without those options.

Suggest fix:

  1. Share the placement helper through utils.py:458–502 and use it in the launch path reached by both entrypoints; update single-controller.md:600 and cluster.md:143 to describe both commands.
  2. Preserve and adapt main's placement tests for the shared launch path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324. main is merged twice in this push (5939fa5 before the review changes, 99ae93a after them), so the branch has no conflicts with main as of 2026-11-16. Both launchers now go through run_single_controller in single_controller_utils/entrypoint_utils.py, which applies controller_placement() (main's _controller_placement, moved) to SingleControllerActor.options(...). main's placement tests are kept in test_entrypoint.py, against the shared path and parametrized over both launchers. docs/cluster.md names both commands, and the simulator guide's Placement section says the controller is placed as in a training run.

Comment thread docs/guides/single-controller.md Outdated
Compaction, mixed image/video conversations, native audio,
video token pruning, static tiling (`image_num_patches`), other processor families,
and `token_capture.defer_routed_experts_to_policy: true` are not supported.
and `token_capture.defer_routed_experts_to_policy: true` / `forest_training: true` are not supported.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forest_training is supported with captured media: rollout_reassembler.py:500–502 explicitly fetches forest media, and test_captured_media.py:1581–1658 checks a successful shared-image forest. This general VLM limitation incorrectly applies the simulator-only restriction to training.

Suggest fix: remove only the forest entry from this VLM bullet.

Suggested change
and `token_capture.defer_routed_experts_to_policy: true` / `forest_training: true` are not supported.
and `token_capture.defer_routed_experts_to_policy: true` are not supported.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 46d7324: the VLM bullet is main's text again. The simulator-only restriction stays in the simulator guide's Limitations.

Comment thread docs/guides/single-controller.md Outdated
What the stand-ins do, per step:

- The rollout pump, `RolloutManager`, `TQReplayBuffer` and the configured sampler run unchanged, so admission is gated exactly as in training: on the trainer version, or on buffer capacity for the `windowed` sampler.
- The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: train/max_seq_mult_prob_error is absent when the supported policy-logprob skip condition in utils.py:24–35 applies; advantage_stage.py:482–486 only records it when that stage runs.

Suggest fix: qualify the copied columns and diagnostic by which logprob stages are required.

Suggested change
- The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`.
- The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into each required policy/reference column. When the policy-logprob stage runs, `train/max_seq_mult_prob_error` reads 1.0; when that stage is skipped, the metric is absent. After all chunks are accepted, it holds `--train-time-s` once. Generation keeps serving during this delay, but forward/backward work overlapping later rollout chunks is not modeled. The delay ends early when rollout activity is idle; adding `train/simulated_train_idle_skipped_s` to `total_step_time` reconstructs this coarse timing model.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 46d7324. The guide now says the CpuTrainer fills each logprob column the recipe requires, that train/max_seq_mult_prob_error reads 1.0 when the policy-logprob stage runs and is absent when it is skipped, and that the hold runs once after all chunks are accepted, so overlap with later chunks of the same step is not modeled.

Comment on lines +115 to +119
uv run tests/check_metrics.py $JSON_METRICS \
'max(data["train/reward"]) > 0' \
'abs(max(data["train/max_seq_mult_prob_error"]) - 1.0) < 1e-6' \
'min(data["train/simulated_train_time_s"]) >= 0' \
'max(data["train/simulated_train_idle_skipped_s"]) <= 15'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: these checks accept a one-step run and never inspect the simulated refit delay, despite the six-step, three-second configuration in grpo_async_gym_single_controller_rollout_only.sh:60–62. The fixed sleep in simulator.py:345–350 runs inside single_controller.py:3578–3589.

Suggest fix: also require all six step metrics and a mean weight-sync time of at least three seconds. Proposed replacement for the metric-check command, not run:

    uv run tests/check_metrics.py $JSON_METRICS \
        'max(data["train/reward"]) > 0' \
        'abs(max(data["train/max_seq_mult_prob_error"]) - 1.0) < 1e-6' \
        'min(data["train/simulated_train_time_s"]) >= 0' \
        'max(data["train/simulated_train_idle_skipped_s"]) <= 15' \
        'len(data["train/simulated_train_time_s"]) == 6' \
        'mean(data["timing/train/weight_sync"]) >= 3'

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324, both checks added. Run on GB200 with vLLM 0.33.0 on the pushed tree: 6 steps, mean timing/train/weight_sync 3.03 s, all six checks pass.

Comment thread docs/guides/single-controller.md Outdated

## Rollout-Only Simulator

`examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the router has an explicit placement rule: setup.py:1102–1106 hard-pins it to the driver's node. This paragraph incorrectly groups it with processes that have no such rule.

Suggest fix: state the router affinity and avoid promising that the unconstrained controller always runs on the head.

Suggested change
`examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run.
`examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver and possibly the `SingleControllerActor`; the generation router, when enabled, is pinned to the driver's node. On a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 46d7324, in the guide's Placement section: the router is pinned to the driver's node; the controller is pinned to the dedicated head when the cluster has one and is otherwise unconstrained.

Comment on lines +245 to +249
# Configured as run_grpo_single_controller.py configures it (same
# skip_tokenizer_init rule, so the engine does the same per-token
# work as one beside a trainer), not as an eval engine. load_format
# is already "auto" from apply_simulator_overrides, and no refit
# delivers draft or MTP weights, so none are expected from one.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: two added comment blocks inside functions exceed the two-line limit in the in-function comment rule.

Suggest fix: shorten both blocks to one line; use these replacements together.

run_single_controller_simulator.py:215–217 replacement:

    # Export telemetry settings before Ray snapshots the worker environment.
Suggested change
# Configured as run_grpo_single_controller.py configures it (same
# skip_tokenizer_init rule, so the engine does the same per-token
# work as one beside a trainer), not as an eval engine. load_format
# is already "auto" from apply_simulator_overrides, and no refit
# delivers draft or MTP weights, so none are expected from one.
# Keep training's tokenizer setup; the engine loads its own checkpoint.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324: the launcher is a thin wrapper now, and the two simulator-specific comments in entrypoint_utils._configure_generation are one line each.

Comment thread docs/guides/single-controller.md Outdated
Comment on lines +598 to +620
## Rollout-Only Simulator

`examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run.

```bash
uv run examples/run_single_controller_simulator.py --config <your-sc.yaml> \
--num-steps 20 --num-nodes 4 --train-time-s 300 --refit-time-s 20 \
<hydra overrides...>
```

The recipe is used as-is and must be an SC recipe (`data_plane.enabled: true`, `async_rl:`, `grpo.async_grpo: null`). The driver overrides only what a rollout-only run owns: `--num-steps` sets `max_num_steps` (and widens `max_num_epochs` so a short dataset cannot clamp it), `--num-nodes` sets the generation node count, `data.validation` is dropped, and `vllm_cfg.load_format` defaults to `auto` so the engine loads the real checkpoint instead of dummy weights plus a first refit (an explicit `dummy` is rejected). Everything else about the engine is configured exactly as `run_grpo_single_controller.py` configures it, including the `skip_tokenizer_init` rule, so it does the same per-token work as an engine beside a trainer.

What the stand-ins do, per step:

- The rollout pump, `RolloutManager`, `TQReplayBuffer` and the configured sampler run unchanged, so admission is gated exactly as in training: on the trainer version, or on buffer capacity for the `windowed` sampler.
- The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`.
- `_sync_weights` runs its full lifecycle -- dispatch gate, `pause_generation_for_refit(clear_cache=recompute_kv_cache_after_weight_updates)`, resume -- around a `NoOpWeightSynchronizer` that moves nothing and holds `--refit-time-s`. The trainer version advances, which releases the sampler's next target.

Everything on the rollout side is available: `generation_router`, `generation_fleet_health` (probes and shard restarts), the stall watchdog, token capture, and the `NRL_NSYS_*` profiling hooks.

`tests/functional/grpo_async_gym_single_controller_rollout_only.sh` is the smoke for this path: the recipe and overrides of `tests/functional/grpo_async_gym_single_controller.sh` (`examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml` on Qwen3-0.6B with a 1-GPU vLLM split) with the trainer stubbed out, gated on a scored rollout and on the stand-in's metrics.

Not supported by `rollout_only`: PPO, non-vLLM generation backends (Megatron generation serves the trainer's own weights; TRT-LLM and SGLang are not validated under the SC refit lifecycle), Dynamo serving (`policy.generation.dynamo: true`, not validated under the simulator's refit lifecycle yet), checkpointing, on-policy distillation, `token_capture.defer_routed_experts_to_policy`, `token_capture.forest_training`, refittable fp8 KV caches, adaptive-ppo-clip, and online quantization (`precision: fp8`, `nvfp4_pertoken_rollout`, `quant_cfg`, `real_quant`): those quantize at refit, which this run has none of. Setup rejects each before any engine starts. The `train_only` mode and the `cpu` rollout backend are tracked in [issue #4860](https://github.com/NVIDIA-NeMo/RL/issues/4860).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The simulator has a separate entrypoint, configuration overrides and timing model; a dedicated guide would keep this page focused on the main SingleController workflow.

Suggest fix: move the new single-controller.md:598–620 section into docs/guides/single-controller-simulator.md, carrying the requested simulator corrections with it. Then leave the short link below and register the guide in docs/index.md:342.

Suggested change
## Rollout-Only Simulator
`examples/run_single_controller_simulator.py` runs one side of the SC loop with the other side mocked, on the same pumps, data plane, sampler and refit lifecycle as a training run. Only `--mode rollout_only` is implemented: generation, the environments (NeMo-Gym or native) and the TransferQueue data plane are real, while the trainer is a `CpuTrainer` (`--train-backend cpu`) that uses no GPU. There is no training cluster: the inference cluster is sized from the recipe's generation node count, or from `--num-nodes`, which gives every node of the allocation to generation. Because every GPU node serves rollouts, a Ray head node with GPUs hosts a vLLM rank next to the driver, the `SingleControllerActor` and the generation router, which have no placement rule of their own; on a training run the head may be a train node instead. Launch from a CPU-only head node, or expect the head's rollout rank to share CPUs with those processes, when comparing against a training run.
```bash
uv run examples/run_single_controller_simulator.py --config <your-sc.yaml> \
--num-steps 20 --num-nodes 4 --train-time-s 300 --refit-time-s 20 \
<hydra overrides...>
```
The recipe is used as-is and must be an SC recipe (`data_plane.enabled: true`, `async_rl:`, `grpo.async_grpo: null`). The driver overrides only what a rollout-only run owns: `--num-steps` sets `max_num_steps` (and widens `max_num_epochs` so a short dataset cannot clamp it), `--num-nodes` sets the generation node count, `data.validation` is dropped, and `vllm_cfg.load_format` defaults to `auto` so the engine loads the real checkpoint instead of dummy weights plus a first refit (an explicit `dummy` is rejected). Everything else about the engine is configured exactly as `run_grpo_single_controller.py` configures it, including the `skip_tokenizer_init` rule, so it does the same per-token work as an engine beside a trainer.
What the stand-ins do, per step:
- The rollout pump, `RolloutManager`, `TQReplayBuffer` and the configured sampler run unchanged, so admission is gated exactly as in training: on the trainer version, or on buffer capacity for the `windowed` sampler.
- The train pump selects the step's groups and runs the advantage stage; the `CpuTrainer` copies `generation_logprobs` into `prev_logprobs` / `reference_policy_logprobs` (so the advantage stage runs unchanged and the training/generation logprob ratio is exactly 1: `train/max_seq_mult_prob_error` reads 1.0), accepts the chunk, and holds `--train-time-s` where `policy.train()` would run. Generation keeps serving during the hold. The hold ends early once no rollout is in flight; `train/simulated_train_idle_skipped_s` reports how much was skipped, so the e2e step time is `total_step_time + simulated_train_idle_skipped_s`.
- `_sync_weights` runs its full lifecycle -- dispatch gate, `pause_generation_for_refit(clear_cache=recompute_kv_cache_after_weight_updates)`, resume -- around a `NoOpWeightSynchronizer` that moves nothing and holds `--refit-time-s`. The trainer version advances, which releases the sampler's next target.
Everything on the rollout side is available: `generation_router`, `generation_fleet_health` (probes and shard restarts), the stall watchdog, token capture, and the `NRL_NSYS_*` profiling hooks.
`tests/functional/grpo_async_gym_single_controller_rollout_only.sh` is the smoke for this path: the recipe and overrides of `tests/functional/grpo_async_gym_single_controller.sh` (`examples/nemo_gym/grpo_qwen3_30ba3b_instruct.yaml` on Qwen3-0.6B with a 1-GPU vLLM split) with the trainer stubbed out, gated on a scored rollout and on the stand-in's metrics.
Not supported by `rollout_only`: PPO, non-vLLM generation backends (Megatron generation serves the trainer's own weights; TRT-LLM and SGLang are not validated under the SC refit lifecycle), Dynamo serving (`policy.generation.dynamo: true`, not validated under the simulator's refit lifecycle yet), checkpointing, on-policy distillation, `token_capture.defer_routed_experts_to_policy`, `token_capture.forest_training`, refittable fp8 KV caches, adaptive-ppo-clip, and online quantization (`precision: fp8`, `nvfp4_pertoken_rollout`, `quant_cfg`, `real_quant`): those quantize at refit, which this run has none of. Setup rejects each before any engine starts. The `train_only` mode and the `cpu` rollout backend are tracked in [issue #4860](https://github.com/NVIDIA-NeMo/RL/issues/4860).
## Rollout-Only Simulator
See the [SingleController simulator guide](single-controller-simulator.md) for setup, usage, timing metrics and limitations.

docs/index.md:342 replacement:

guides/single-controller.md
guides/single-controller-simulator.md

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324: the section moved to docs/guides/single-controller-simulator.md, registered in docs/index.md, with the corrections from the other threads. The main guide keeps the short link.

Comment thread examples/run_grpo_single_controller.py Outdated
setup_single_controller,
)
from nemo_rl.algorithms.single_controller_utils.utils import (
run_with_controller_liveness_watch,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could you help to add a new nemo_rl/algorithms/single_controller_utils/watchdog.py, and move WatchdogConfig and run_with_controller_liveness_watch to there instead of mix things in utils.py?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324: WatchdogConfig and run_with_controller_liveness_watch are in single_controller_utils/watchdog.py, and utils.py is back to main's content.

_DEFAULTS = SimulatorSettings()


def parse_args() -> tuple[argparse.Namespace, list[str]]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we move it to a yaml to follow the existing behavior in the repo?

for the simulator, I think we can load two config yaml, one is the simulator yaml to keep new things in simulator, another is the train config just like before. since we might need to add more configs later, I think it's not clear keeping the configs just in examples/run_single_controller_simulator.py, yaml is more straightforward to show.

@terrykong wdyt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 46d7324, in the shape you describe. --config is the SC recipe, loaded as before. --simulator-config (default examples/configs/simulator/rollout_only.yaml) holds the simulator keys and is validated by SimulatorConfig, a pydantic model with extra="forbid" that replaces the SimulatorSettings dataclass and the six flags. Overrides keep the Hydra syntax: simulator.<key>=<value> addresses the simulator YAML, anything else the recipe. test_config_validation.py checks the shipped YAML against the schema. Happy to change the shape if terrykong prefers another one.

…nly-simulator

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

# Conflicts:
#	docs/guides/single-controller.md
#	examples/run_grpo_single_controller.py
#	nemo_rl/algorithms/single_controller_utils/utils.py
#	tests/unit/single_controller/test_entrypoint.py
… settings to YAML

Signed-off-by: Shiki Wu <shikiw@nvidia.com>
…nly-simulator

Signed-off-by: Shiki Wu <shikiw@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants